What Is Prometheus? OSS Monitoring & Alerting Toolkit

Prometheus is an open-source monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, it has grown into a robust monitoring system and, in August 2018, became a graduated project of the Cloud Native Computing Foundation (CNCF).

In monitoring, Prometheus is used for collecting and processing metrics from its targets, which can be applications, servers, databases, or any system that exposes metrics via an HTTP endpoint. It uses a pull-based model, where it periodically scrapes the targets' metrics endpoints to gather data on various aspects such as resource utilization, response times, error rates, and custom application metrics. This data is then stored in a time-series database and can be queried using PromQL (Prometheus Query Language), enabling users to analyze trends, visualize data in dashboards, and set up alerts based on predefined conditions.

How Prometheus Works

Prometheus uses a pull-based model: the Prometheus server scrapes metrics over HTTP from the /metrics endpoints exposed by your targets, at a configured interval. It stores each sample in its local time-series database (TSDB), where every series is identified by a metric name and a set of key-value labels. You can query this data with PromQL, and Prometheus evaluates alerting rules against it, sending any alerts that fire to Alertmanager. Here's how each part works:

Pull-based Scraping

The Prometheus server requests metrics from configured targets, such as applications, databases, and microservices, over HTTP or HTTPS. Each target exposes a metrics endpoint, commonly /metrics, which Prometheus scrapes at a configured interval. Applications can expose these metrics through client libraries, while exporters make metrics available from systems that do not expose them directly.

Time-series storage

Prometheus stores the collected measurements locally in its time-series database (TSDB). Each time series is identified by a metric name and a set of key-value labels, with each sample recording a value and timestamp. For example, labels on a request counter can distinguish requests by HTTP method or response status, so you can examine each group separately.

Service discovery

Targets can be listed in the configuration file or found through service discovery. In Kubernetes, Prometheus can discover targets as workloads change. With file-based discovery, it watches target lists maintained in files. This reduces the need to update the main configuration whenever a monitored instance changes.

PromQL queries

Prometheus Query Language (PromQL) lets you filter, aggregate, and calculate values from stored metrics. For example, rate(http_requests_total[5m]) estimates the average requests per second over the previous five minutes for each matching time series. You can use queries to investigate a change in traffic, compare services, or define alert conditions.

Alerting

Prometheus evaluates alerting rules written with PromQL at configured intervals. A rule might check whether a service's error rate stays above a threshold for five minutes. When the alert fires, Prometheus sends it to Alertmanager, which groups related alerts, applies silences and inhibition rules, and routes notifications to receivers such as Slack, email, or PagerDuty.

Visualization

The built-in Prometheus web UI displays query results as tables or graphs, which is useful for checking a metric or investigating a problem. For reusable dashboards, Grafana can query Prometheus as a data source. SigNoz Cloud offers another way to collect these metrics. The OpenTelemetry Collector scrapes Prometheus-format metrics from your targets and sends them to SigNoz Cloud for storage and visualization. We'll cover this setup later in the article.

How to Set Up Prometheus

This quick start guide shows how to run Prometheus locally and check that it can collect metrics from itself. The commands below use a Linux or macOS shell.

Step 1: Download Prometheus

Visit the Prometheus download page and select the package for your operating system and CPU architecture. Download a stable release and extract the archive. On Windows, extract the ZIP archive and run prometheus.exe instead of ./prometheus.

Step 2: Configure the prometheus.yml file

Open prometheus.yml in the extracted directory. For this example, use the following configuration to scrape Prometheus itself every 15 seconds:

global:
  scrape_interval: 15s
 
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

The targets entry points to the running Prometheus server. You can add more scrape jobs later for your applications and exporters.

Step 3: Start the Prometheus server

From the extracted directory, check the configuration and start Prometheus:

./promtool check config prometheus.yml
./prometheus --config.file=prometheus.yml

Keep the process running while you use the web UI. Prometheus listens on port 9090 by default.

Step 4: Access the Prometheus UI

Open http://localhost:9090/ in your browser. After at least one scrape interval, enter up{job="prometheus"} in the query field and click Execute. A value of 1 means the last scrape succeeded; 0 means it failed.

The UI screenshots below were captured with Prometheus 3.8.0 on port 19090 because the default port was already in use. Your instance will show localhost:9090 when you follow the configuration above. To use another port, change both the scrape target and the server's --web.listen-address setting.

Open Status → Target health, or visit http://localhost:9090/targets, to inspect the scrape endpoint. The prometheus target should show UP.

Core Metric Types in Prometheus

Prometheus client libraries offer four core metric types. Choose one based on what you measure.

• Counter - A counter accumulates events, such as HTTP requests or errors. It increases during normal operation and can reset to zero when the application restarts. Use rate() to calculate its per-second rate of increase.

• Gauge - A gauge represents a value that can rise or fall, such as current memory usage or requests in progress. The difference between gauges and counters determines which fits your measurement.

• Histogram - A histogram records a distribution of observations, such as request durations, using buckets. Classic histograms expose cumulative bucket counts, a sum, and a count; native histograms use composite samples with dynamic buckets. Our guide to histogram buckets explains classic bucket boundaries. PromQL can estimate percentiles from histogram data.

• Summary - A summary records observation counts and sums and can calculate configured quantiles in the application over a sliding time window. Unlike histogram-derived percentiles, these precomputed quantiles cannot be meaningfully aggregated across instances.

  For example, averaging the p95 latencies reported by two application instances does not give the service's combined p95. For that calculation, aggregate histogram data first, then estimate the percentile with histogram_quantile(). Classic histograms need compatible bucket boundaries. This distinction also matters when a StatsD exporter translates application timers into Prometheus metrics: choose a histogram representation when you need percentiles across instances.

Key Components of Prometheus Architecture

A Prometheus deployment is built around the Prometheus server, which scrapes and stores metrics. The other components help it find targets, expose metrics from your applications and third-party systems, handle short-lived jobs, and route alerts.

Prometheus Server

The Prometheus server is the core component responsible for scraping and storing time-series data. It periodically scrapes metrics from configured targets and stores the collected samples locally. It also processes PromQL queries and evaluates recording and alerting rules.

Pushgateway

The Pushgateway is an optional intermediary that allows selected service-level batch jobs to push metrics to it for Prometheus to scrape. This is useful when a job finishes before Prometheus can scrape it directly. Pushgateway retains pushed metrics until they are explicitly deleted, so you need to manage stale metrics.

Exporters

Exporters are used to expose metrics from third-party systems, such as databases, hardware, or other services, in a format that Prometheus can scrape. Exporters act as bridges between Prometheus and the systems being monitored, converting their native metrics into a Prometheus-compatible format.

Alertmanager

Alertmanager handles alerts generated by the Prometheus server. It deduplicates and groups alerts, then routes notifications to receivers such as email, PagerDuty, or Slack. Silences mute notifications for matching alerts during a specified period, while inhibition suppresses notifications when another firing alert already explains the problem.

Client Libraries

Client libraries are available in various programming languages, including Go, Java, and Python. These libraries enable developers to instrument their applications and expose metrics in a format that Prometheus can scrape. This allows custom application metrics to be integrated into the monitoring system.

Prometheus Targets

Prometheus targets are the endpoints from which the Prometheus server scrapes metrics. These targets can be instrumented applications, services, or systems that expose metrics endpoints. Targets can be specified manually or discovered dynamically through service discovery mechanisms.

Service Discovery

Service discovery helps Prometheus find targets in dynamic environments such as Kubernetes and Consul. The Prometheus server then scrapes the discovered endpoints according to its scrape configuration. This keeps the target list aligned with changes in your infrastructure without requiring you to edit each target manually.

Benefits of Prometheus monitoring

Prometheus helps you investigate problems, monitor changing infrastructure, and define alerts using the metrics you collect.

• Query metrics by service, instance, or status. Labels and PromQL let you narrow down a problem. For example, you can compare error rates across services or check whether one instance is handling more traffic than the others.
• Keep monitoring independent. Each Prometheus server collects, stores, and queries its own data without requiring a separate storage service. This helps keep monitoring available when other parts of your infrastructure fail.
• Monitor changing infrastructure. Service discovery helps Prometheus keep track of targets in environments such as Kubernetes, reducing the need to maintain endpoint lists manually.
• Monitor existing systems through exporters. Exporters expose metrics from databases, operating systems, and other services, so you don't have to build every integration yourself.
• Define alerts around your metrics. PromQL expressions let you specify the conditions that need attention. Alertmanager can group related alerts and route notifications to the appropriate team.
• Use it without software licence fees. Prometheus is open source. You still need to account for infrastructure, storage, and the work involved in operating it.

Monitor Prometheus metrics with SigNoz Cloud

SigNoz Cloud is a managed observability platform built on OpenTelemetry that brings metrics, logs, and traces into one place. You can use it to visualize your Prometheus metrics, build dashboards, and set up alerts without running the monitoring backend yourself.

If your application or exporter already exposes metrics at a /metrics endpoint, the OpenTelemetry Collector's Prometheus receiver can scrape that endpoint and send the metrics to SigNoz Cloud over OTLP. You can keep your existing instrumentation, and this collection path does not require a Prometheus server between the endpoint and the Collector.

Get started with SigNoz Cloud and follow the Prometheus metrics integration guide to configure collection. Once your metrics appear in Metrics Explorer, you can use them to build dashboards and configure alerts.

If you prefer to run the platform on your own infrastructure, you can self-host SigNoz. Follow the SigNoz self-host installation guide to get started.