Is Prometheus Monitoring Push or Pull?

Prometheus uses a pull-based model for monitoring. In this model, the Prometheus server regularly pulls metrics from configured targets by scraping their endpoints. This contrasts with a push-based model where the monitored systems send (push) their metrics to the monitoring server. Pull-based monitoring is preferred for its flexibility, centralized control, and security advantages.

How Pull-Based Monitoring Works in Prometheus

From the above architecture diagram, Prometheus scrapes or pulls metrics from the targets configured for it. Targets are the applications or services that expose metrics endpoints.

Metrics endpoints are specific URLs exposed by applications or services where Prometheus can access the metrics data. These endpoints provide a standardized format of metrics that Prometheus can scrape and store. The metrics offer detailed insights into the performance, resource usage, and operational health of the applications or services.

Targets can be discovered in several ways: statically configured, dynamically discovered through service discovery mechanisms, or manually configured. Each target is identified by its URL and accompanied by a set of labels that provide descriptive information about the target.

Example of target configuration:

scrape_configs:
  - job_name: 'prometheus' 
    static_configs: 
      - targets: ['prometheus:9090']
      
  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
    - targets: ['cadvisor:8080']

  - job_name: 'mysqld_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['mysqld_exporter:9104']

  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['node-exporter:9100']

The above Prometheus configuration file defines several scrape jobs for Prometheus to collect metrics from different targets at specified intervals: The targets being scraped here are:

• Prometheus: Prometheus is configured to monitor itself by scraping its own metrics endpoint (prometheus:9090).
• cAdvisor: cAdvisor is a container monitoring tool that provides metrics about container resource usage. Prometheus is configured to scrape metrics from cAdvisor (cadvisor:8080) every 5 seconds.
• mysqld_exporter: This exporter provides metrics about a running MySQL database server. Prometheus is configured to scrape metrics from mysqld_exporter (mysqld_exporter:9104) every 5 seconds.
• node_exporter: This exporter provides hardware and operating system metrics from the host machine. Prometheus is configured to scrape metrics from node_exporter (node-exporter:9100) every 5 seconds.

The above image shows the /metrics endpoint of the cAdvisor container at:8000/metrics which exposes metrics about containers on a system. Prometheus server initiates HTTP requests to the /metrics endpoints exposed by the applications or services being monitored.

In the Prometheus targets UI, you can see the targets being monitored, their current status, and details about the last scrape. This interface provides an overview of all configured scrape targets, indicating whether they are up or down, the last scrape duration, and any errors encountered during scraping.

Once Prometheus pulls the metrics from the endpoints, it stores them in its time series database which can be queried for visualization and monitoring using the Prometheus web UI or a monitoring tool.

Can Prometheus do Push-Based Monitoring?

Yes, Prometheus can also do push-based monitoring using the Pushgateway. The Prometheus Pushgateway allows you to push time series from short-lived service-level batch jobs to an intermediary job which Prometheus can scrape. This is useful for monitoring components that cannot be scraped.

Example of using Pushgateway:

echo "some_metric 3.14" | curl --data-binary @- http://localhost:9091/metrics/job/some_job

In this example, a metric is pushed to the Pushgateway, which Prometheus then scrapes.

Limitations of Push-Based Monitoring with the Pushgateway

While Prometheus primarily relies on a pull-based model to collect metrics, it does offer the Pushgateway as an alternative mechanism. However, the Prometheus documentation strongly advises using the Pushgateway only in specific, limited scenarios. This is due to several drawbacks that can arise when using the Pushgateway instead of the standard pull model for general metric collection:

• Single Point of Failure: When monitoring multiple instances through a single Pushgateway, it can become a single point of failure and a potential performance bottleneck.
• Loss of Instance Health Monitoring: By using the Pushgateway, you lose Prometheus's automatic instance health monitoring via the up metric, which is generated during each scrape.
• Persistent Data Exposure: The Pushgateway retains all series pushed to it and will continue to expose them to Prometheus indefinitely unless manually deleted via the Pushgateway's API.

Pull-Based Monitoring vs Push-Based Monitoring

There are clear differences between the pull-based and push-based monitoring approach with Prometheus.

Pull-Based Monitoring

In pull-based monitoring, Prometheus actively scrapes metrics from configured targets at regular intervals. This method relies on Prometheus having knowledge of which targets it should scrape and how often. Here's how it works:

• Configuration: You define what to scrape (targets) and how often (scrape interval) in the Prometheus configuration file (prometheus.yml).
• Scraping: At each scrape interval, Prometheus sends HTTP requests to the specified endpoints to collect metrics.
• Storage: Collected metrics are stored in Prometheus for querying and alerting purposes.

Push-Based Monitoring

Push-based monitoring, on the other hand, involves targets pushing their metrics to Prometheus. This approach is useful when dealing with a large number of ephemeral targets or monitoring targets that cannot be scraped.

Here's how push-based monitoring works:

• Exposition: Targets expose their metrics via HTTP endpoints, typically using the Prometheus exposition format.
• Pushgateway: For short-lived jobs or batch processes without persistent storage, the Pushgateway can temporarily store metrics before they are scraped by Prometheus.
• Relabeling: Prometheus uses relabeling rules to determine which pushed metrics to ingest based on labels attached to the metrics.

Pros and Cons of Pull and Push Based Monitoring

In this section, we will discuss the pros and cons of the push and pull based monitoring model Prometheus uses.

Pros of Pull-Based Monitoring

• Simplicity: Pull-based monitoring is straightforward to set up and manage since Prometheus actively scrapes metrics from defined targets at scheduled intervals. You also don't need to install agents on target systems to push metrics.
• Service Discovery: Prometheus easily integrates with service discovery mechanisms (like Kubernetes, Consul, etc.) to automatically discover and scrape new targets as they appear, maintaining monitoring coverage.
• Reliability: Prometheus can detect when a target is down, making it easier to identify and troubleshoot outages.
• Control: You have fine-grained control over how often and which metrics are scraped from each target.

Cons of Pull-Based Monitoring

• Scalability: Pull-based systems can face scalability issues as the number of targets increases, requiring more resources to manage.
• Complexity: Managing and maintaining Prometheus configuration files (prometheus.yml) for target discovery and scraping rules, can be complex, especially in large-scale deployments.
• Resource Consumption: Frequent scraping can increase the load on both the monitored services and the Prometheus server, potentially impacting performance.
• Potential Downtime: If Prometheus goes down or there's a network issue, there might be a delay in metric collection until the next scheduled scrape.

Pros of Push-Based Monitoring

• Reduced Load on Services: By having targets push metrics, the load on monitored services is reduced since they only report metrics when available, not on every scrape cycle.
• Flexibility: Ideal for short-lived jobs and environments where targets cannot expose endpoints for scraping, as it doesn't require the jobs to run long enough to be scraped by a monitoring system. The job itself initiates the sending of metrics, making it ideal for transient environments.
• Efficiency in Resource Utilization: Since metrics are sent only when available, this method can be more efficient in terms of network and CPU usage.

Cons of Push-Based Monitoring

• Potential for Metric Loss: There's a risk of missing metrics if the target is unavailable at the moment Prometheus attempts to scrape, especially in highly dynamic environments.
• Dependency on External Systems: For short-lived jobs or batch processes, the reliance on external systems like the Pushgateway introduces another component that could fail or become a single point of failure.

Use Cases for Pull-Based and Push-Based Monitoring

Knowing when to use the pull-based and push-based monitoring approach in Prometheus is important. Here are some use cases for each of them”

Pull-Based Monitoring

1. Dynamic Infrastructure Monitoring: In dynamic environments like Kubernetes, services can scale up and down dynamically. Prometheus can automatically discover and scrape metrics from these services, providing up-to-date monitoring without manual intervention.
2. Long-Lived Services: For applications and services that run continuously, such as web servers and databases, the pull model ensures consistent and reliable metrics collection.

Push-Based Monitoring

1. Short-Lived Jobs: For batch jobs or ephemeral processes that start and complete quickly, pushing metrics to the Pushgateway ensures that their performance data is not lost.
2. Highly Secure Environments: In environments where targets are behind firewalls or have limited outbound network access, it may be easier to push metrics to a central location rather than exposing endpoints for scraping.

Conclusion

This guide outlines how Prometheus does pull-based monitoring as well as push-based monitoring. It also covers the pros and cons of each monitoring approach as well their use cases.

Some Key takeaways include:

• Prometheus primarily uses a pull-based monitoring approach.
• Prometheus can do push-based monitoring through the Pushgateway.
• The Pushgateway has certain limitations and should be used for scenarios where the component to be monitored cannot be scraped.
• The pull-based monitoring approach is ideal for dynamic environments and long-lived services, while the push-based approach is ideal for short-lived or ephemeral jobs.