Skip to main content

Monitor HAProxy Metrics and Logs with OpenTelemetry [Step By Step Guide]

· 13 min read
Abhishek Kothari

For extremely high throughput web applications, it is important to load balance the traffic across multiple servers. However, load balancing the traffic alone is not enough at times. The reverse proxy server that handles the workload needs to be performant, too. In our previous article, we discussed the NGINX reverse proxy server and understood how to monitor it. In this article, we set up monitoring for an even more performant reverse proxy server - HAProxy.

Cover Image

In this tutorial, we cover:

If you want to jump straight into implementation, start with this prerequisites section.

A Brief Overview of HAProxy‚Äč

HAProxy proudly calls itself a reliable and highly performant load-balancer. It does just one job - and does it very well. It is an open-source server available as a load balancer for TCP and HTTP traffic. It is, in particular, used for high-traffic websites due to its performant algorithms for load-balancing requests. A single HAProxy server has been proven to serve over 2M requests per second with SSL enabled. A server this performant requires excellent monitoring, too. It does come with an inbuilt dashboard for showing its metrics, but it isn’t as intuitive as it should be. In this article, we talk about how to achieve good monitoring capabilities around HAProxy, including metrics and logs using OpenTelemetry and Signoz.

A Brief Overview of OpenTelemetry‚Äč

OpenTelemetry is a set of APIs, SDKs, libraries, and integrations aiming to standardize the generation, collection, and management of telemetry data(logs, metrics, and traces). It is backed by the Cloud Native Computing Foundation and is the leading open-source project in the observability domain.

The data you collect with OpenTelemetry is vendor-agnostic and can be exported in many formats. Telemetry data has become critical in observing the state of distributed systems. With microservices and polyglot architectures, there was a need to have a global standard. OpenTelemetry aims to fill that space and is doing a great job at it thus far.

In this tutorial, you will use OpenTelemetry Collector to collect HAProxy metrics and logs for monitoring.

What is OpenTelemetry Collector?‚Äč

OpenTelemetry Collector is a stand-alone service provided by OpenTelemetry. It can be used as a telemetry-processing system with a lot of flexible configurations to collect and manage telemetry data.

It can understand different data formats and send it to different backends, making it a versatile tool for building observability solutions.

Read our complete guide on OpenTelemetry Collector

How does OpenTelemetry Collector collect data?‚Äč

A receiver is how data gets into the OpenTelemetry Collector. Receivers are configured via YAML under the top-level receivers tag. There must be at least one enabled receiver for a configuration to be considered valid.

Here’s an example of an otlp receiver:

receivers:
otlp:
protocols:
grpc:
http:

An OTLP receiver can receive data via gRPC or HTTP using the OTLP format. There are advanced configurations that you can enable via the YAML file.

Here’s a sample configuration for an otlp receiver.

receivers:
otlp:
protocols:
http:
endpoint: "localhost:4318"
cors:
allowed_origins:
- http://test.com
# Origins can have wildcards with *, use * by itself to match any origin.
- https://*.example.com
allowed_headers:
- Example-Header
max_age: 7200

You can find more details on advanced configurations here.

After configuring a receiver, you must enable it. Receivers are enabled via pipelines within the service section. A pipeline consists of a set of receivers, processors, and exporters.

The following is an example pipeline configuration:

service:
pipelines:
metrics:
receivers: [otlp, prometheus]
exporters: [otlp, prometheus]
traces:
receivers: [otlp, jaeger]
processors: [batch]
exporters: [otlp, zipkin]

Pre-requisites‚Äč

This tutorial assumes that you have the OpenTelemetry collector installed on the same host as your HAProxy if you plan to monitor a local HAProxy setup. In case you are looking to monitor a remote HAProxy setup, you would need to open up the network port to allow the OpenTelemetry collector to access the metrics.

Logs, however, would require an OpenTelemetry collector to run locally to access the files.

Preparing HAProxy‚Äč

HAProxy comes pre-installed for most popular Linux distributions. However, in case you do not have it installed, you can download it from here or install it by following the below instructions.

Installing HAProxy on MacOS

You can easily install HAProxy on MacOS using HomeBrew.

brew update && brew install haproxy

Installing HAProxy on Ubuntu

You can install HAProxy on Ubuntu using apt

sudo apt install --no-install-recommends software-properties-common

sudo add-apt-repository ppa:vbernat/haproxy-2.4 -y

sudo apt update && sudo apt install haproxy -y

Installing HAProxy on other Linux distros using yum

yum update && yum install haproxy

Once installed, we can configure HAProxy to contain an endpoint showing the server statistics. In order to do so, modify /etc/haproxy/haproxy.conf file as shown below.

We are intentionally binding the frontend to 8000 proxy to avoid conflict with any other server running on port 80.

You would also require a backend server running behind the scenes to allow HAProxy to forward the traffic.

Note here that the status URL configured is localhost:8000/haproxy?stats .

global
log /dev/log local0
---

---

---
frontend haproxy-main
bind *:8000
option forwardfor
stats uri /haproxy?stats
default_backend node_webservers
backend node_webservers
balance roundrobin
server local <your-local-webserver-url> check

Next, for logs, modify the file /etc/rsyslog.d/49-haproxy.conf (file name may differ) and add the below lines.

$AddUnixListenSocket /var/lib/haproxy/dev/log
# Send HAProxy messages to a dedicated logfile
:programname, startswith, "haproxy" {
/var/log/haproxy.log
stop
}

Your HAProxy server is ready to serve the metrics and logs now.

For the purpose of this tutorial, we will assume the following:

  • HAProxy is installed on the same host when you are setting up OpenTelemetry collector
  • Logs for HAProxy will be enabled and stored on the path /var/log/haproxy.log

Once configured, run the below commands:

sudo service rsyslog restart
sudo service haproxy restart

You should start seeing stats on http://localhost:8000/haproxy?stats as shown below.

HAProxy stats on your local
HAProxy stats on your local

Setting up OpenTelemetry Collector‚Äč

Step 1 - Downloading OpenTelemetry Collector‚Äč

Download the appropriate binary package for your Linux or macOS distribution from the OpenTelemetry Collector releases page. We are using the latest version available at the time of writing this tutorial.

For MACOS (arm64):

curl --proto '=https' --tlsv1.2 -fOL https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.90.0/otelcol-contrib_0.90.0_darwin_arm64.tar.gz

Step 2 - Extracting the package‚Äč

Create a new directory named otelcol-contrib and then extract the contents of the archive into this newly created directory with the following command:

mkdir otelcol-contrib && tar xvzf otelcol-contrib_0.90.0_darwin_arm64.tar -C otelcol-contrib

Step 3 - Setting up the configuration file‚Äč

In this tutorial, we would not just be monitoring the metrics of HAProxy but also visualizing its access and error logs. Let’s create a config.yaml in the otelcol-contrib folder. This will configure the collector with the HAProxy receiver for metrics and File Log receiver for logs.

Note:

The configuration file should be created in the same directory where you unpack the otel-collector-contrib binary. If you have globally installed the binary, it is okay to create on any path.

receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317
http:
endpoint: localhost:4318
haproxy:
endpoint: http://127.0.0.1:8000/haproxy?stats
metrics:
filelog:
include:
- /var/log/haproxy.log

processors:
batch:
send_batch_size: 1000
timeout: 10s
attributes:
actions:
- key: app
value: haproxy
action: insert
exporters:
otlp:
endpoint: "ingest.{region}.signoz.cloud:443" # replace {region} with your region
tls:
insecure: false
headers:
"signoz-access-token": "{your-signoz-token}" # Obtain from https://{your-signoz-url}/settings/ingestion-settings
logging:
verbosity: detailed
service:
telemetry:
metrics:
address: localhost:8888
pipelines:
metrics:
receivers: [otlp, haproxy]
processors: [batch]
exporters: [otlp]
logs:
receivers: [filelog]
processors: [attributes]
exporters: [otlp]

You would need to replace region and signoz-token in the above file with the region of your choice (for Signoz Cloud) and token obtained from Signoz Cloud ‚Üí Settings ‚Üí Integration Settings.

You can find ingestion details in the SigNoz dashboard
You can find ingestion details in the SigNoz dashboard

The above configuration is quite simple - We utilize the local HAProxy setup exposing statistics on http://localhost:8000/haproxy?stats URL. You can also monitor multiple HAproxy servers by adding multiple receivers, as shown below:

receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317
http:
endpoint: localhost:4318
haproxy:
endpoint: http://127.0.0.1:8000/haproxy?stats
metrics:
haproxy/2:
endpoint: http://<remote-url>/haproxy?stats
metrics:
filelog:
include:
- /var/log/haproxy.log

processors:
batch:
send_batch_size: 1000
timeout: 10s
attributes:
actions:
- key: app
value: haproxy
action: insert
exporters:
otlp:
endpoint: "ingest.{region}.signoz.cloud:443" # replace {region} with your region
tls:
insecure: false
headers:
"signoz-access-token": "{your-signoz-token}" # Obtain from https://{your-signoz-url}/settings/ingestion-settings
logging:
verbosity: detailed
service:
telemetry:
metrics:
address: localhost:8888
pipelines:
metrics:
receivers: [otlp, haproxy, haproxy/2]
processors: [batch]
exporters: [otlp]
logs:
receivers: [filelog]
processors: [attributes]
exporters: [otlp]

Step 4 - Running the collector service‚Äč

Every Collector release includes an otelcol executable that you can run. Since we’re done with configurations, we can run the collector service with the following command.

From the otelcol-contrib, run the following command:

./otelcol-contrib --config ./config.yaml

If you want to run it in the background -

./otelcol-contrib --config ./config.yaml &> otelcol-output.log & echo "$\!" > otel-pid

Step 5 - Debugging the output‚Äč

If you want to see the output of the logs, you may look it up with:

tail -f -n 50 otelcol-output.log

tail 50 will give the last 50 lines from the file otelcol-output.log

You can stop the collector service with the following command:

kill "$(< otel-pid)"

You should start seeing the metrics on your Signoz Cloud UI in about 30 seconds.

Monitoring with SigNoz Dashboard‚Äč

You can import this dashboard JSON into your SigNoz environment quite easily to monitor your HAproxy instance.

Once the above setup is done, you will be able to access the metrics in the SigNoz dashboard. You can go to the Dashboards tab and try adding a new panel. You can learn how to create dashboards in SigNoz here.

HAProxy metrics collected by OpenTelemetry collector
HAProxy metrics collected by OpenTelemetry collector

You can easily create charts with query builder in SigNoz. Here are the steps to add a new panel to the dashboard.

Creating a panel for Network in traffic - Bytes per second
Creating a panel for Network in traffic - Bytes per second

You can build a complete dashboard around various metrics emitted. Here’s a look at a sample dashboard we built out using the metrics collected.

Comprehensive HAProxy dashboard
Comprehensive HAProxy dashboard

You can also create alerts on any metric. Learn how to create alerts here.


Visualizing HAProxy logs‚Äč

In order to visualize the logs sent by OpenTelemetry collector, head over to Signoz ‚Üí Logs ‚Üí Logs Explorer. In the logs explorer, you can filter your logs using the tag app=haproxy, as shown in the image below:


You can visualize the volume of logs as well as the actual log lines easily in this UI.

Reference: HAProxy metrics collected by OpenTelemetry Collector‚Äč

Below HAProxy metics are enabled by default to be collected by OpenTelemetry Collector:

MetricDescriptionType
haproxy_bytes_inputIncoming Network trafficSum
haproxy_bytes_outputOutgoing Network trafficSum
haproxy_connections_errorsTotal Connection errorsSum
haproxy_connections_rateRate of connectionsGauge
haproxy_connections_retriesTotal connection retriesSum
haproxy_requests_deniedTotal requests deniedSum
haproxy_requests_errorsTotal request errorsSum
haproxy_requests_queuedTotal requests in queueSum
haproxy_requests_rateNet request rateGauge
haproxy_requests_redispatchedTotal requests redispatchedSum
haproxy_requests_totalTotal number of requests handledSum
haproxy_responses_deniedTotal number of responses deniedSum
haproxy_responses_errorsTotal errors in responsesSum
haproxy_server_selected_totalNumber of times server was selected when redispatching the requestSum
haproxy_sessions_averageAverage session timeGauge
haproxy_sessions_countTotal active sessionsGauge
haproxy_sessions_rateNumber of sessions per secondGauge

The below metrics need to be explicitly enabled for haproxyreceiver:

MetricDescriptionType
haproxy_clients_canceledNumber of data transfers aborted by the clientSum
haproxy_compression_bypassNumber of bytes bypassing the compression engineSum
haproxy_compression_countNumber of HTTP responses that got compressedSum
haproxy_compression_inputTotal bytes fed into the compressorSum
haproxy_compression_outputTotal bytes emitted out by compressorSum
haproxy_connections_totalTotal connections to the frontendSum
haproxy_downtimeBackend downtimeSum
haproxy_failed_checksTotal failed backend checksSum
haproxy_sessions_totalTotal HAProxy sessions created during the lifetimeSum

Attributes‚Äč

All the above metrics have the below three attributes associated:

NameDescriptionValuesEnabled
haproxy_addraddress:port or "unix". IPv6 has brackets around the address.Any Strtrue
haproxy_proxy_nameProxy nameAny Strtrue
haproxy_service_nameService name (FRONTEND for frontend, BACKEND for backend, any name for server/listener)Any Strtrue

Conclusion‚Äč

In this tutorial, you installed an OpenTelemetry Collector to collect HAProxy metrics and logs and sent the collected data to SigNoz for monitoring and alerts.

Visit our complete guide on OpenTelemetry Collector to learn more about it. OpenTelemetry is quietly becoming the world standard for open-source observability, and by using it, you can have advantages like a single standard for all telemetry signals, no vendor lock-in, etc.

SigNoz is an open-source OpenTelemetry-native APM that can be used as a single backend for all your observability needs.


Further Reading‚Äč