How to Handle Null Values in Prometheus Time Series Data

When working with Prometheus time series data, encountering gaps or missing data points can be frustrating. These absences can disrupt visualizations and skew your analysis. However, setting these missing data points (or gaps) to zero can help maintain continuity in your dashboards, though it’s essential to understand when to apply this solution to avoid misleading interpretations.

In this article, we’ll explore why gaps occur in Prometheus data, how to handle them using PromQL, and when it's appropriate to fill them with zero. We'll also touch on how Grafana and SigNoz can enhance your ability to manage missing data. Let’s dive in.

Understanding Missing Data in Prometheus Time Series

In Prometheus, a "null value" doesn’t exist in the typical database sense. Instead, when we talk about missing data, we mean the absence of a data point at a specific time. This absence shows up as a gap in your time series graphs.

Common Causes of Missing Data:

1. Scrape Failures: Prometheus scrapes targets at regular intervals. If a target is down or unreachable, Prometheus can’t gather data, leaving a gap.
2. Absent Metrics: Sometimes, metrics are not emitted during a certain period, especially in applications that only report specific events.
3. Counter Resets: For counter metrics (which track cumulative counts), resets can appear as a missing data point or a sudden drop, depending on how the graph is visualized.

Why Does This Matter?

These missing data points can confuse visualizations. You might misinterpret gaps as zero or simply overlook them in key parts of your analysis. Properly handling missing data ensures that your graphs reflect reality without introducing misleading trends. You can understand this better with the help of the scenario listed below.

Scenario: Monitoring Server CPU Usage

Imagine you're monitoring CPU usage across multiple servers in your infrastructure using Prometheus and visualizing the data in Grafana. The metric you're tracking is cpu_usage_percent, which records the percentage of CPU utilization at regular intervals (every 10 seconds).

Normal Situation

When everything is running smoothly, the graph in Grafana shows a continuous trend of CPU usage, fluctuating between 30% and 80%:

Time:   |--------|--------|--------|--------|
CPU %:  |  40%   |  50%   |  70%   |  65%   |

You can easily spot high usage peaks and take necessary action, such as scaling resources if the usage remains consistently high.

Data Gaps Without Proper Handling

Now, imagine that for a few minutes, one of your servers stops sending metrics due to a network issue or because Prometheus couldn't scrape the data. This results in missing data points during that period:

Time:   |--------|--------|--------|--------|--------|--------|
CPU %:  |  40%   |  50%   |   -    |   -    |   -    |  65%   |

• Impact: If the graph doesn't handle missing data well, it will show gaps where the data is missing. You might overlook this gap and assume that nothing significant happened during that time. However, it could represent a critical outage, a CPU spike, or other vital information that went unnoticed due to missing data.

Should You Set Gaps as Zero in Prometheus?

When is Setting Gaps to Zero a Good Idea?

There are cases where filling in missing data points with zero can make sense, especially when:

• You’re analyzing event activity: Some applications only create an event counter after the first event occurs. If the application is being scraped successfully and the absence means no events have occurred, a zero can make those inactive periods easier to follow.
• You want consistent graphs: When missing values represent known inactivity, filling them with zeros helps distinguish those periods from periods with recorded activity. A collection failure should still appear as missing data.

When Should You Avoid Setting Gaps to Zero?

Filling in missing data with zero isn’t always the right solution. For certain metric types, especially counters, setting a gap to zero can suggest something that didn't actually happen:

• Counter metrics: Zero might imply a reset or sudden drop, which could skew your understanding of system behavior.
• Sporadic metrics: For metrics that are emitted only under specific conditions, check what an absent value means. For example, a missing measurement of how long a task took does not mean the task took zero seconds.

For example, consider the missing CPU usage data in the previous scenario. You might configure the graph to default missing data to zero. Now, your graph looks like this:

Time:   |--------|--------|--------|--------|--------|--------|
CPU %:  |  40%   |  50%   |   0%   |   0%   |   0%   |  65%   |

• Impact: The graph now shows that CPU usage dropped to 0% for those missing intervals. This can lead to a misleading trend where it seems like your server experienced zero CPU activity during that period. However, the reality is that no data was collected, not that the CPU usage dropped to zero. You might falsely conclude that everything was fine or under-utilized, when in fact, the server might have been down or highly overloaded.

Filling gaps can also change your calculations. If two servers report 20% and 40% CPU usage, their average is 30%. Including a third server with an assumed value of zero lowers the average to 20%, even though its CPU usage is unknown.

Methods to Handle Missing Data in Prometheus Queries

Once you know whether a gap represents inactivity or a collection problem, you can choose how to handle it. Start with a zero fallback for expected inactivity or use an absence check when the metric should be present. For metrics with several labeled series, the fallback also needs to match those labels.

Basic Methods for Handling Missing Data

1. Using or vector(0)

   PromQL provides vector(0) to create a single zero-valued sample with no labels. Combining it with a metric using or keeps the metric's values and adds the zero if no result has the same labels:

   Copy

   custom_metric or vector(0)

   If custom_metric has no labels, this returns its value when present and zero when absent. If the metric has labels such as job and instance, the query adds a separate unlabeled zero series. It doesn't fill each missing labeled series.

   

   For a single total across the metric's series, you can remove the labels with sum() before applying the fallback:

   Copy

   sum(custom_metric) or vector(0)

   This returns the total when values are available and zero when no series is present. For example, if custom_metric counts events and is only created after the first event, this can show zero before any events have occurred, provided collection is working. The advanced example below covers keeping separate results for each target.

2. Utilizing the absent() Function

   • When a metric should always be present, use absent() to detect when it stops reporting.
   Copy

   absent(custom_metric)

   • Explanation: This query returns 1 when no custom_metric series is present and no result when at least one series exists. This is useful for alerting on a metric that should always be reported.

   

   Use Case: In a microservices architecture, you'll want to know if a service’s health metric stops being scraped. absent() can alert you that the service may be down or unreachable, helping to detect system failures early.

   If several instances report this metric, the query only detects when all of them are absent. Add an instance selector when checking a particular target.

Advanced PromQL Techniques for Null Handling

1. Handling Label Matching When Merging Zero Values

   The basic or vector(0) example provides one zero without labels. To keep a separate value for each target, you need a fallback that identifies the targets you expect. Prometheus's up metric provides their job and instance labels.

   Suppose custom_metric is an event counter with only job and instance labels, and an absent counter means no events have occurred. You can supply zeros for successfully scraped targets with this query:

   Copy

   custom_metric{job="my_service"}
   or on (job, instance)
   (0 * (up{job="my_service"} == 1))

   The right side selects targets with successful scrapes and multiplies their values by zero. The or on (job, instance) operation keeps existing counter values and adds a labeled zero for each of those targets that has no counter yet. If one target reports a count of 12 and another hasn't had an event, the result shows 12 for the first and zero for the second. A failed scrape doesn't provide a zero fallback.

   This works when job and instance identify the series you need. If the event metric also has labels such as a route or error type, up cannot supply those missing combinations.

   The group_left and group_right modifiers are useful for matching different numbers of series when combining two metrics. They don't supply missing values, so they aren't needed for this zero fallback.

   When a metric is expected to be present, filling its absence with zero could hide a reporting problem. The next method checks whether that absence continues over a time window.

2. Using absent_over_time() for Detecting Long-Term Absences

   • Scenario: You’re monitoring a critical metric over time and want to detect if it has stopped reporting for an extended period, indicating a problem with the service or the metric’s availability.
   • Solution: Use absent_over_time() to monitor whether a metric has been missing for a defined period.
   Copy

   absent_over_time(custom_metric[1m])

   Explanation: This function returns 1 if there are no samples for custom_metric anywhere in the last minute. If any matching series has a sample in that window, it returns no result. Choose a window that allows for the metric’s expected reporting interval.

   

Best Practices for Handling Null/Missing Values in Time Series Data

To effectively manage null values in the Prometheus time series:

• Understand the Causes: Investigate why data is missing, such as network issues or service downtime, and document these findings for future reference.
• Choose Appropriate Methods: Set missing values to zero when they represent known inactivity. Use absent() or absent_over_time() when a metric should be present, and match labels when providing a fallback for individual series.
• Regularly Audit Data Collection: Monitor the metrics collection process to ensure it functions correctly. Adjust scrape intervals based on data volatility.
• Document Strategies: Maintain clear documentation on how missing values are handled, and use comments in PromQL queries for clarity.

Monitoring Gaps in Data with SigNoz

SigNoz is a powerful observability tool that allows you to track missing data points more effectively, helping you avoid blind spots in your monitoring. SigNoz can help you in:

• Detecting Missing Data with SigNoz SigNoz can alert you to missing data automatically by configuring rules for your most critical metrics. This means you won’t have to rely on visually spotting gaps in your Grafana dashboards. Instead, SigNoz can proactively notify you.
• Visualizing Data Gaps in SigNoz Dashboards SigNoz provides advanced visualizations for analyzing patterns in missing data. This can help you see trends where certain metrics go missing frequently, helping you dig into root causes and improve data collection reliability.

SigNoz Cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 24,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

Key Takeaways

• Missing data points (or gaps) in Prometheus time series can occur because of scrape failures, application restarts, or metrics that have not yet been emitted.
• Filling missing data with zero can show expected inactivity, but it can also hide collection failures and change averages.
• Use or vector(0) for an unlabeled zero fallback, a matching labeled fallback for individual series, and absent() to detect missing metrics.
• Best practices involve choosing appropriate methods, documenting strategies, and regular audits of null handling approaches.
• Tools like SigNoz can provide advanced capabilities for monitoring, alerting, and analyzing null value patterns in your time series data.

FAQs

What causes null values in Prometheus time series data?

Missing values in Prometheus time series can be caused by temporary service outages, network issues, scrape failures, or the absence of specific metrics during certain time intervals.

How do null values affect time series analysis?

Null or missing values can lead to broken graphs, skewed calculations, and false alerts. They can make trend analysis difficult and potentially lead to misinterpretation of data.

Can setting nulls to zero lead to data misinterpretation?

While setting nulls/missing values to zero can improve data continuity, it's important to understand the context of your metrics. For some types of data, zero might not be an appropriate replacement for a null value. Always consider the nature of your metrics when deciding how to handle null values.

What are the differences in handling nulls for counters vs. gauges?

For counters, inserting zero into a gap can make the graph look as though the counter reset. For gauges such as CPU usage, zero can imply inactivity when the measurement is actually unknown. In either case, use zero only when you know the absence means no activity, such as an event counter that has not yet been created.

Why does or vector(0) add a separate zero series?

The zero from vector(0) has no labels. When your metric has labels, or keeps both the labeled metric and the unmatched zero. To fill individual series, use a fallback with the labels you expect, as shown in the label-matching example.