Overview
When a metric has a volume control rule, some of its attributes are aggregated away to reduce stored cardinality and cost, and queries over older time ranges render from that aggregated data. The data you're looking at is mathematically correct, but it's stored at a coarser granularity than the original. For some questions, that changes the answer.
This page explains exactly what is preserved, what is lost, and why a query can return different values than it did when the data was fresh.
What Aggregation Stores
When a rule is active, all series of a metric that differ only in the aggregated-away attributes are collapsed into one series, in fixed 60-second windows. For each window, each surviving series stores:
- Gauges: the minimum and maximum across all original values in the window, the sum of each original series' last value, the sum of all values, and the number of samples and of contributing series. This is enough to answer min, max, sums of current values, and both kinds of average (see below).
- Counters and histograms (counts, sums, and buckets): the reset-aware increase for the window, how much each original series actually grew, summed across the collapsed series, correctly accounting for counter resets and process restarts.
What is not stored: the aggregated-away attributes (by design), the individual values within the window, and their timing inside the minute.
The Timing Model
- The most recent 24 hours are always full fidelity. Rules only affect how data is stored beyond this window. Any query whose range falls entirely inside it sees every attribute and every raw sample, exactly as ingested.
- Alerts always evaluate on full-fidelity data. Alert evaluation runs over the recent full-fidelity window, never over aggregated data.
- Aggregated data trails real time by a few minutes (typically 2–4), because aggregation runs on a schedule after data settles. You'll normally never notice, since the freshest data is served from the full-fidelity window anyway.
The full-fidelity window exists for a reason: the moments where every attribute matters are alert evaluation and live incident debugging, and both happen on recent data. By guaranteeing the last 24 hours raw, alerting stays reliable regardless of what rules exist, and a severe outage can be investigated immediately at full detail with no need to touch volume control rules or weigh the cost of keeping detail you might need. What aggregation trades away is detail in history, where the questions are usually about trends rather than individual instances.
A Worked Example
Say queue.depth is a gauge reported every 15 seconds by three instances of a service, and a rule aggregates away service.instance.id. During the minute starting 12:00:00 the raw samples were:
| Reported at | instance A | instance B | instance C | Sum across instances |
|---|---|---|---|---|
| 12:00:15 | 10 | 5 | 20 | 35 |
| 12:00:30 | 12 | 5 | 18 | 35 |
| 12:00:45 | 50 | 6 | 15 | 71 |
| 12:01:00 | 8 | 5 | 22 | 35 |
After aggregation, one series holds one row for this minute: min 5, max 50, sum of last values 35 (8 + 5 + 22), sum of all values 176, 12 samples, 3 series.
Now compare what common queries return on raw vs aggregated data:
| Question | On raw data | On aggregated data |
|---|---|---|
| Total queue depth (sum of instances' latest values) | 35 | 35 |
| Highest single reading in the minute | 50 | 50 |
| Number of reporting instances | 3 | 3 |
| Peak total queue depth (max over time of the sum) | 71 at 12:00:45 | not computable (closest available is the group max, 50) |
| Sum of each instance's peak (sum of per-series max) | 78 (50 + 6 + 22) | not computable (collapses to the group max, 50) |
| Which instance spiked to 50? | instance A | not answerable (the attribute is gone) |
The pattern: aggregation had to commit to one per-window summary per statistic. Queries that match a stored statistic are exact. Queries that combine statistics in a different order, like "sum of maxes" or "max of sums", can't be reconstructed, because they need the individual series and sample timings back. Sum-of-last and sum-of-max only agree when each series reports once per window, or when its last value happens to be its maximum.
Two Kinds of Average
"Average" hides two different questions, and aggregation keeps both, but you should know which one your panel asks:
- Spatial average (the average across instances, now): sum of last values / number of series. In the example: 35 / 3 ≈ 11.7.
- Temporal average (the average level over the window): sum of all values / number of samples. In the example: 176 / 12 ≈ 14.7.
Both are answerable from aggregated data. But a subtler form, "average of each instance's average" (weighting each instance equally regardless of how often it reports), is not, and silently becomes the pooled temporal average when instances report at different rates.
Questions That Become Unanswerable
Once an attribute is aggregated away for a time range, for that range you can no longer:
- Filter or group by it. "Requests by pod", "top 5 instances by memory", "errors excluding host X": none of these can be answered, for any query over the aggregated range.
- Identify outliers. You can see that some instance hit 50 (the max survives); you cannot see which one, or whether it was the same instance each minute.
- Count series above a threshold. Only the count of contributing series per window survives, not their individual values.
- See sub-minute detail. A 15-second spike keeps its magnitude (it's the window max) but loses its exact timing and shape. Rates finer than 60 seconds are no longer meaningful.
Everything expressible over the surviving attributes at 60-second resolution (including exact counter rates and increases) remains correct.
Why an Alert's Numbers May Not Reproduce Later
Alerts evaluate on full-fidelity data. Investigations often happen days later, on aggregated data. The same query can legitimately return different values in those two views.
Continuing the example: an alert on total queue depth > 60 fired at 12:00:45, when the instantaneous sum across instances hit 71. Two weeks later you investigate, and the panel, now rendering from aggregated data, plots the per-minute sum of last values: 35. The line never crosses 60. The alert was not wrong, and the data is not corrupted; the 71 existed between the per-minute summaries that were kept.
When an investigation on aggregated data doesn't reproduce what an alert saw, check whether the alert's query depends on within-window timing (max of a sum, instantaneous thresholds) or on attributes the rule aggregated away.
What You Can Do
- Query within the full-fidelity window. Investigations within 24 hours of the data being written see everything, unaggregated.
- Keep the attributes you alert and group by. Review the related-assets warning when creating a rule. It lists dashboards and alerts that query the attributes you're about to aggregate away.
- Adjust the rule if you got it wrong. Removing an attribute from a rule restores full fidelity for it going forward. Already-aggregated history stays aggregated.
