What Is a Prometheus Target? Configuration and Troubleshooting

A Prometheus target is an endpoint or resource that exposes metrics for collection. These targets serve as the primary data sources for Prometheus monitoring, allowing Prometheus to gather valuable information about your systems and applications.

Prometheus targets come in various forms, including:

• Applications instrumented with Prometheus client libraries
• Servers exposing system-level metrics
• Databases offering performance statistics
• Third-party systems with Prometheus-compatible metric endpoints

This guide explains how Prometheus discovers targets, how to configure their metrics endpoints, and how to add labels that identify each service or host. It also covers authentication, custom HTTP headers, and troubleshooting failed scrapes.

How Prometheus discovers targets

Prometheus needs a list of targets to scrape. You can define this list in two ways:

• Static configuration: Here, you add target addresses under static_configs in prometheus.yml. This is useful when you have a small set of targets whose addresses rarely change.
• Service discovery: In service discovery, Prometheus gets target addresses from Kubernetes, DNS, cloud APIs, or YAML/JSON files. If you use files to list your targets, prometheus watches them for changes and updates its target list automatically, without needing to reload prometheus.yml.

How to configure Prometheus targets

To configure a target, you need an endpoint that exposes metrics and a scrape job that points to it. After adding the job, validate the configuration and reload prometheus to apply it. If your application already serves metrics, you can skip to the scrape job configuration and use its address instead.

Set up a metrics endpoint

Prometheus collects metrics from an HTTP endpoint, such as http://localhost:8000/metrics. To configure this endpoint, add localhost:8000 to targets in your scrape configuration. Prometheus uses /metrics as the default path, so you only need to set metrics_path if your application serves metrics at a different path.

If your application doesn’t expose metrics yet, you can add a Prometheus client library. The Python example below runs a simple loop and uses a counter to track how many times it completes. In your own application, you’d increment the counter after the operation you want to track finishes.

First, create a virtual environment and install the library. On macOS or Linux, run:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install prometheus-client

Save this code as metrics_demo.py:

import time
from prometheus_client import Counter, start_http_server
 
completed_tasks = Counter('demo_tasks_completed_total', 'Total completed demo tasks')
 
if __name__ == '__main__':
    start_http_server(8000, addr='127.0.0.1')
    while True:
        time.sleep(1)
        completed_tasks.inc()

Run the script with python3 metrics_demo.py. Then, open another terminal and check the metrics endpoint:

curl -fsS http://127.0.0.1:8000/metrics

You’ll see several metrics in the output. Look for these lines:

# HELP demo_tasks_completed_total Total completed demo tasks
# TYPE demo_tasks_completed_total counter
demo_tasks_completed_total 5.0

The counter increases once per second, so you may see a different value depending on when you run curl.

The HELP line describes the metric, and TYPE tells Prometheus it’s a counter. The final line gives the metric’s name and current value. The client library formats this output for you, so you only need to define your metrics and update them in your code.

Configure the scrape job

Now that the metrics endpoint is running, add a scrape job to prometheus.yml so Prometheus knows where to collect the metrics.

Add the following under scrape_configs in your prometheus.yml. If your configuration already has a scrape_configs section, add the job to the existing list rather than creating a second section.

scrape_configs:
  - job_name: 'python-demo'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets: ['127.0.0.1:8000']

This job tells Prometheus to scrape http://127.0.0.1:8000/metrics every 15 seconds.

Validate, reload, and check the target

Before applying the changes, check your configuration for errors:

promtool check config prometheus.yml

If the check passes, reload Prometheus to pick up the changes. You can do this by sending SIGHUP to the Prometheus process. If you started Prometheus with --web.enable-lifecycle, you can also reload it with an HTTP request:

curl -fsS -X POST http://localhost:9090/-/reload

Restarting Prometheus also applies the changes, but reloading avoids stopping the server.

After reloading, open the target health page to check whether Prometheus can scrape the endpoint. You can also run this query:

up{job="python-demo"}

A value of 1 means the scrape succeeded; 0 means it failed.

Finally, query the counter from the Python example to confirm Prometheus is collecting it:

demo_tasks_completed_total{job="python-demo"}

Scrape targets that require authentication or custom headers

Some metrics endpoints require credentials or extra headers with each request. You can configure these in the scrape job so Prometheus sends them when collecting metrics.

For example, this job connects to an HTTPS endpoint that requires a tenant ID and an API key. The http_headers setting adds both headers to each scrape request:

scrape_configs:
  - job_name: 'protected-app'
    scheme: https
    metrics_path: /metrics
    http_headers:
      X-Tenant-ID:
        values: ['team-a']
      X-API-Key:
        files: ['/etc/prometheus/secrets/metrics-api-key']
    static_configs:
      - targets: ['app.example.com:443']

Replace the targets, header names, and tenant ID with the values your endpoint expects. Save your API key in the specified file and make sure the Prometheus process can read it. If Prometheus runs in a container, mount the file at that path inside the container.

These headers are sent to every target in the job. If your targets need different credentials, put them in separate scrape jobs.

Use values for non-sensitive headers and files to load credentials from a separate file. You can also use secrets for sensitive values written directly in the configuration. This hides them on the configuration page, but they remain in the YAML file. Keep credentials out of version control and send them over HTTPS.

For endpoints that use standard authentication methods, use Prometheus’s dedicated settings instead of adding an Authorization header under http_headers:

• HTTP Basic authentication: Use basic_auth with username and password_file.
• Bearer tokens: Use authorization with type: Bearer and credentials_file. The file should contain only the token, without the Bearer prefix.

Once you’ve made the changes, validate and reload the configuration, then check the target health page. If the endpoint returns 401 or 403, check that the credentials are correct, the header names match what it expects, and the credentials grant access to the metrics endpoint.

Restrict access to metrics endpoints

Metrics can reveal internal service names and details about your infrastructure. In production, keep metrics endpoints on private networks. Use firewall rules or Kubernetes NetworkPolicies to limit access to your Prometheus servers.

Keep credentials and personal data out of metric labels. Also avoid values such as user IDs, which can create a large number of time series.

If your endpoint needs authentication or TLS, configure it in the application, exporter, or a reverse proxy in front of it. The scrape-job settings in the previous section tell Prometheus how to authenticate. However, they don’t protect the endpoint itself.

For endpoints that require mutual TLS, use tls_config in the scrape job with the appropriate ca_file, cert_file, and key_file. Keep certificate verification enabled.

Add target-specific labels

Labels help you identify which environment, service, or region a target belongs to. For static targets, add them under labels in static_configs:

scrape_configs:
  - job_name: 'api-server'
    static_configs:
      - targets: ['api-1.example.com:8080']
        labels:
          environment: 'production'
          service: 'api'
          region: 'us-west-2'
      - targets: ['api-staging.example.com:8080']
        labels:
          environment: 'staging'
          service: 'api'
          region: 'us-west-2'

Each set of labels applies to every target in the same targets list. If targets need different label values, put them in separate groups, as shown above.

Validate and reload the configuration, then check the labels on the target health page. You can now select the production target with:

up{job="api-server", environment="production"}

Before adding a label, check whether your application already exposes one with the same name. By default, Prometheus keeps the configured target label and adds an exported_ prefix to the conflicting label from the scraped metric. Setting honor_labels: true reverses this behavior. The scraped label keeps its value instead.

For targets found through service discovery, you can use relabel_configs to turn discovery metadata into labels. For example, add these rules to an existing Kubernetes pod scrape job:

relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod

These rules copy the Kubernetes namespace and pod name into namespace and pod labels, making them available in queries. Labels starting with __meta_ are temporary and are removed after target relabeling unless you copy their values.

Target relabeling runs before scraping. It differs from metric_relabel_configs, which processes scraped samples, and from PromQL's label_replace(), which changes query results. Choose stable label values and avoid high-cardinality values such as request IDs. Our Prometheus labels guide explains label selection in more detail.

Use a hostname as the instance label

By default, Prometheus sets the instance label to the target’s host and port, such as 192.0.2.10:9100 or worker-01:9100. If you scrape an IP address but want a readable name in your queries, add a hostname label and copy its value to instance:

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['192.0.2.10:9100']
        labels:
          hostname: 'worker-01'
      - targets: ['192.0.2.11:9100']
        labels:
          hostname: 'worker-02'
    relabel_configs:
      - source_labels: [hostname]
        regex: '(.+)'
        target_label: instance

Prometheus still scrapes 192.0.2.10:9100, but the collected metrics now have instance="worker-01" and hostname="worker-01". The rule changes the label without changing __address__, which tells Prometheus where to send the request. The regex skips targets with an empty or missing hostname label.

If your target list changes often, you can keep the addresses and hostnames in a separate file and reuse the same rule. For example, save this as /etc/prometheus/targets/nodes.yml:

- targets: ['192.0.2.10:9100']
  labels:
    hostname: 'worker-01'
- targets: ['192.0.2.11:9100']
  labels:
    hostname: 'worker-02'

Then replace static_configs in the scrape job with the following, keeping the relabel_configs rule from above:

file_sd_configs:
  - files: ['/etc/prometheus/targets/nodes.yml']

Prometheus picks up changes to this file automatically. You still need to supply the hostname for each target. Relabeling doesn’t look up hostnames through reverse DNS.

Keep instance unique within each job. If you run several exporters on the same host, include a port or another distinguishing value so they don’t share the same instance label. Changing a label value creates new time series for future scrapes, it doesn’t rename historical data. Check any dashboards or alerts that use the old instance values before making the change. If you’d rather leave those queries untouched, keep instance as it is and use the separate hostname label for filtering or grouping.

Troubleshooting Prometheus target issues

Start with the target health page and its last scrape error. Test connectivity from the host or container running Prometheus, since an endpoint reachable from your laptop may not be reachable from the scraper.

up = 1 confirms a successful scrape, not the health of every part of the application. An absent up series can mean the target is no longer discovered or was dropped. It is different from a failed scrape returning up = 0.

Monitor target metrics with SigNoz Cloud

SigNoz Cloud is an all in one observability platform for metrics, logs, and traces. You can collect metrics from your existing Prometheus-format endpoints with the OpenTelemetry Collector's Prometheus receiver and send them to SigNoz Cloud. This lets you reuse the metrics your applications already expose.

Follow the Prometheus metrics integration guide to configure the scrape jobs in the Collector and verify that metrics arrive. These settings belong in the Collector's configuration; editing a standalone Prometheus server's prometheus.yml does not send data to SigNoz Cloud. Once data arrives, use it to build dashboards and configure metric alerts.

Start with SigNoz Cloud, or self-host SigNoz if you want to run it on your own infrastructure.

FAQs

What’s the difference between a Prometheus target and an exporter?

A target is the endpoint Prometheus scrapes. An exporter collects metrics from another system and exposes them at an endpoint Prometheus can scrape. If your application already exposes Prometheus-compatible metrics, you don’t need a separate exporter.

How often does Prometheus scrape targets by default?

Prometheus’s built-in default is one minute, though many example configurations set scrape_interval to 15s. Each job uses the global interval unless it defines its own, so check both settings to see how often your targets are scraped.

Can Prometheus automatically discover new targets in a Kubernetes cluster?

Yes. Use kubernetes_sd_configs to discover targets through the Kubernetes API. Choose the discovery role that matches what you want to monitor, grant Prometheus the necessary API permissions, and use target relabeling to select the endpoints and ports to scrape.

How many targets can a single Prometheus instance handle?

There’s no single limit that applies to every setup. It depends on how many time series your targets produce, how often you scrape them, your query workload, and the CPU, memory, and storage available. Test with your expected workload to work out the capacity you need.