What are Prometheus labels - Key Concepts and Best Practices

Last Updated: October 30, 202611 min read

Prometheus labels are essential components of the Prometheus monitoring system, playing a crucial role in how metrics are organized and queried. These key-value pairs provide context and additional information to metrics, enabling powerful and flexible data analysis. This article explores the concept of Prometheus labels, their significance in the Prometheus ecosystem, and best practices for their effective use.

What are Prometheus Labels: Understanding the Basics

Prometheus labels are key-value pairs attached to metrics that provide additional context and characterization. They form the foundation of Prometheus's dimensional data model, allowing for the creation of multi-dimensional time series data. Unlike metrics — which represent measurable values — labels offer metadata that describe the specific instance or attributes of a metric.

For example, consider a metric http_requests_total. Without labels, this metric would only show the total number of HTTP requests. However, by adding labels such as method="GET" and status="200", you can create distinct time series for different types of requests:

http_requests_total{method="GET", status="200"} 1234
http_requests_total{method="POST", status="404"} 5

This labeling system allows for more granular and meaningful data analysis, enabling you to slice and dice your metrics based on various dimensions.

How Labels Enhance Prometheus Metrics

Labels transform Prometheus metrics from simple counters or gauges into rich, multi-dimensional data points. They create unique time series for each combination of labels, allowing for flexible querying and filtering. This capability is particularly powerful when combined with Prometheus's query language, PromQL.

Consider a real-world scenario: monitoring a web application across multiple servers. Without labels, you might have separate metrics for each server:

server1_cpu_usage 0.75
server2_cpu_usage 0.60
server3_cpu_usage 0.80

With labels, you can consolidate this into a single metric with server-specific labels:

cpu_usage{server="server1"} 0.75
cpu_usage{server="server2"} 0.60
cpu_usage{server="server3"} 0.80

This approach simplifies metric management and enables powerful queries. For instance, you can easily find the average CPU usage across all servers or identify the server with the highest usage.

Understanding the Job and Instance Labels

When monitoring several servers, you may want to look at one server or at all the servers running the same service. Prometheus adds job and instance labels to scraped metrics to help you make that distinction. By default, job identifies the group, while instance identifies an individual target by its host and port.

For example, two servers running an API can belong to the same scrape job. You configure both Prometheus targets under one job_name:

scrape_configs:
  - job_name: 'api-servers'
    static_configs:
      - targets: ['api1.example.com:8080', 'api2.example.com:8080']

Both targets receive job="api-servers". Their instance values are api1.example.com:8080 and api2.example.com:8080, so Prometheus can keep their readings separate within the job.

To check whether Prometheus can scrape these targets, select the job's up metric:

up{job="api-servers"}

The result has one series per target. A value of 1 means its last scrape succeeded, while 0 means it failed. The instance label tells you which target each result belongs to.

Why Prometheus Labels Matter: Benefits and Importance

Prometheus labels are not just a convenient feature — they are fundamental to the system's effectiveness. Here's why they matter:

Fine-grained data analysis: Labels allow you to drill down into specific aspects of your metrics, enabling detailed analysis and troubleshooting.
Multi-dimensional metrics: By using labels, you can represent complex, multi-faceted data points without creating separate metrics for each dimension.
Improved organization: Labels provide a systematic way to categorize and classify metrics, making your monitoring system more manageable and scalable.
Enhanced PromQL capabilities: Labels unlock the full potential of PromQL, allowing for complex queries that can aggregate, filter, and transform data based on label values.

Best Practices for Using Prometheus Labels

To maximize the benefits of Prometheus labels while avoiding potential pitfalls, follow these best practices:

Choose meaningful and consistent label names: Use clear, descriptive names that reflect the purpose of the label. Stick to a consistent naming convention across your metrics.
Manage label cardinality: Be mindful of the number of unique label value combinations. High cardinality can lead to performance issues and increased storage requirements.
Use labels to improve metric discoverability: Implement labels that make it easier to find and understand metrics, such as application, environment, or team.
Standardize label usage: Establish guidelines for label usage across your organization to ensure consistency and ease of use.

Querying Prometheus Label Values

Label filters let you focus on a particular server, environment, or operation. For example, to examine successful GET requests, select the method and status values from the request metric:

http_requests_total{method="GET", status="200"}

This returns the series that match both conditions, along with their labels and current counter values. You can use other matchers to exclude values or select a pattern:

Matcher	Meaning	Example
`=`	Equals	`method="GET"`
`!=`	Does not equal	`method!="GET"`
`=~`	Matches a regular expression	`status=~"4..\|5.."`
`!~`	Does not match a regular expression	`status!~"5.."`

Grouping Metrics by Labels

Filtering helps you examine a specific set of series. To compare activity across HTTP methods, you can group the series and calculate a request rate for each method:

sum by (method) (rate(http_requests_total[5m]))

Here, rate() calculates requests per second over the last five minutes. sum by (method) adds those rates across instances and status codes, giving you one result for each method. Only the method label remains in the result.

To see how successful and failed requests contribute to those rates, keep the status label as well:

sum by (method, status) (rate(http_requests_total[5m]))

Each method and status combination now has its own result. For example, the rate of GET requests with status 200 is separate from the rate with status 500.

The by clause works well when you know which labels you want to keep. If you want to combine readings across instances while retaining all the other labels, use without instead:

sum without (instance) (rate(http_requests_total[5m]))

This removes instance from the result and combines the rates where the remaining labels match. Labels such as job, method, and status stay available for comparing the results.

Grouping by a Derived Label

Sometimes the grouping you need is part of an existing label value. For example, the API job's instance values contain a hostname and port. Use label_replace() to copy the hostname into a host label and group request rates by it:

sum by (host) (
  label_replace(
    rate(http_requests_total{job="api-servers"}[5m]),
    "host", "$1", "instance", "([^:]+):.*"
  )
)

For instance="api1.example.com:8080", $1 captures api1.example.com and sets it as the host value. The query returns one request rate per hostname, combining readings across ports, methods, and status codes. label_replace() changes labels in query results without changing the stored series.

Listing and Counting Unique Label Values

Grouping can also help you find which label values exist in a metric. If you want to list the methods rather than calculate their request rates, use group by:

group by (method) (http_requests_total{method!=""})

This returns one series for each method, with a value of 1. The method!="" filter excludes series where the method label is missing or empty.

To get the number of distinct methods, count the groups:

count(group by (method) (http_requests_total{method!=""}))

If the metric contains GET and POST series, the result is 2. Having several instances or status codes for the same method does not increase that count. The result reflects the methods present when the query runs.

You may also want to know how many series belong to each method. For that, use count by:

count by (method) (http_requests_total{method!=""})

Three GET series and one POST series produce a count of 3 for GET and 1 for POST. These counts describe the series, not the number of requests they recorded. The similarly named count_values() counts sample values rather than label values.

Getting Label Values Through the HTTP API

The queries above return series with labels attached. If you need label names or a list of values for a script or dashboard dropdown, use the Prometheus HTTP API.

To retrieve the label names:

curl 'http://localhost:9090/api/v1/labels'

To retrieve the method values for the request metric:

curl --get 'http://localhost:9090/api/v1/label/method/values' \
  --data-urlencode 'match[]=http_requests_total'

The response's data field contains the values, such as ["GET", "POST"]. The match[] selector restricts the lookup to http_requests_total. Without that selector, the endpoint lists method values across metrics.

Limitations of Prometheus Labels

More label combinations increase resource usage. Each unique combination creates another time series, adding to memory usage, storage requirements, and the work needed to query a metric.

For example, a request metric with 3 methods, 5 status codes, 100 endpoints, and 2 instances can produce 3,000 series if every combination occurs. Adding a user_id label can increase that number as more users make requests. A metric with many unique label combinations has high cardinality.

To keep this number manageable, use route templates such as /users/:id instead of raw paths such as /users/123. Requests for different users then share the same route value. Keep details about individual users or requests in logs or traces.
Grouping query results does not reduce stored series. Queries can help you identify metrics with many series, but they do not reduce the data Prometheus already stores. If resource usage is higher than expected, start by finding the metrics with the most series in the API job:
```
topk(10, count by (__name__) ({job="api-servers"}))
```
This query counts series for each metric name and returns the ten largest counts. Inspect the labels on those metrics to find values that keep growing, then review whether you need that detail.
Dropping unwanted samples can remove useful readings. Adjusting labels in the application can prevent unnecessary series from being created. If you do not need readings that carry a particular label, you can exclude them before storage with metric_relabel_configs:
```
scrape_configs:
  - job_name: 'api-servers'
    static_configs:
      - targets: ['api1.example.com:8080', 'api2.example.com:8080']
    metric_relabel_configs:
      - source_labels: [user_id]
        regex: '.+'
        action: drop
```
This configuration drops every scraped sample with a non-empty user_id label. It removes the whole sample, not just that label. If you still need an overall request count, your application should expose an aggregate metric without the user ID.

Advanced Concepts: Relabeling in Prometheus

Relabeling is a powerful feature in Prometheus that allows you to modify, add, or remove labels before they are stored. This process occurs during scraping and can be used to:

Standardize label names
Filter out unwanted metrics
Add additional context to metrics

Relabeling is configured using the relabel_config block in the Prometheus configuration file. Here's a basic example:

scrape_configs:
  - job_name: 'example'
    static_configs:
      - targets: ['example.com:8080']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '(.*):.*'
        replacement: '$1'

This configuration extracts the hostname from the __address__ label and sets it as the instance label.

FAQs

What is the difference between a metric and a label in Prometheus?

A metric in Prometheus represents a measurable value, such as the number of requests or CPU usage. Labels, on the other hand, provide additional context or metadata for these metrics. They allow you to create multiple time series for a single metric based on different label combinations.

How do labels affect Prometheus performance?

Labels can significantly impact Prometheus performance, particularly when dealing with high cardinality. Each unique combination of labels for a metric creates a new time series, which increases storage requirements and query complexity. Carefully managing label cardinality is crucial for maintaining optimal performance.

Can I add or modify labels after data collection?

While you cannot directly modify labels on existing data points, Prometheus provides several ways to work with labels post-collection:

Relabeling during scraping
Using recording rules to create new metrics with modified labels
Applying label transformations during querying using PromQL functions

What is label cardinality and why is it important?

Label cardinality refers to the number of unique label value combinations for a given metric. High cardinality can lead to performance issues and increased storage requirements. It's important to balance the need for detailed metrics with the potential impact on system resources.

Monitoring with SigNoz Cloud

While Prometheus is a powerful monitoring solution, managing labels and ensuring optimal performance can be challenging. SigNoz Cloud offers a comprehensive monitoring platform that builds upon Prometheus's strengths while addressing its limitations.

SigNoz Cloud provides:

Efficient handling of high-cardinality data
User-friendly interface for exploring and visualizing metrics
Advanced querying capabilities without the complexity of PromQL
Integrated tracing and logging for a complete observability solution

To get started with SigNoz Cloud:

SigNoz Cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 24,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

By using SigNoz Cloud, you can leverage the power of Prometheus-style metrics and labels while benefiting from additional features and ease of use. Whether you're using SigNoz Cloud or the open-source version, SigNoz offers a robust solution for modern monitoring needs.