Traefik Observability with OpenTelemetry

Overview

In this tutorial, we will see how to export metrics and traces of Traefik to SigNoz. Visualizing Traefik metrics and traces will help you to understand the performance of services running behind Traefik and troubleshoot issues.

Prerequisites

• Traefik v3.0 or above
• SigNoz setup (choose one):
  • SigNoz Cloud account with an active ingestion key
  • Self-hosted SigNoz instance

Steps

To send Traefik metrics and traces to SigNoz, you need to configure Traefik to export data using the OpenTelemetry Protocol (OTLP).

Step 1: Configure Traefik CLI Flags

You need to enable OTLP export for both metrics and traces in your Traefik configuration.

For Metrics:

• --metrics.otlp=true: Enables OTLP metrics export.
• --metrics.otlp.grpc=true: Uses gRPC for transport (recommended).
• --metrics.otlp.grpc.endpoint=ingest.{region}.signoz.cloud:443: Sets the destination endpoint.
• --metrics.otlp.grpc.headers.signoz-ingestion-key=SIGNOZ_INGESTION_KEY: Authenticates with your SigNoz Cloud key.

For Traces:

• --tracing.otlp=true: Enables OTLP tracing.
• --tracing.otlp.grpc=true: Uses gRPC for transport.
• --tracing.otlp.grpc.endpoint=ingest.{region}.signoz.cloud:443: Sets the destination endpoint.
• --tracing.otlp.grpc.headers.signoz-ingestion-key=SIGNOZ_INGESTION_KEY: Authenticates with your SigNoz Cloud key.

Step 2: Create the Docker Compose File

We will use a docker-compose.yaml file to run Traefik and a sample hello-app service. This configuration sets up Traefik as a reverse proxy and configures it to send telemetry data to SigNoz.

Create a file named docker-compose.yaml with the following content:

version: '3'
services:
  reverse-proxy:
    image: traefik:latest
    extra_hosts:
      - signoz:host-gateway
    command:
      - --api.insecure=true
      - --providers.docker
      # Metrics Configuration
      - --metrics.otlp=true
      - --metrics.otlp.grpc=true
      - --metrics.otlp.grpc.insecure=false
      - --metrics.otlp.grpc.endpoint=ingest.{region}.signoz.cloud:443
      - --metrics.otlp.grpc.headers.signoz-ingestion-key=SIGNOZ_INGESTION_KEY
      # Tracing Configuration
      - --tracing.otlp=true
      - --tracing.otlp.grpc=true
      - --tracing.otlp.grpc.insecure=false
      - --tracing.otlp.grpc.endpoint=ingest.{region}.signoz.cloud:443
      - --tracing.otlp.grpc.headers.signoz-ingestion-key=SIGNOZ_INGESTION_KEY
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  hello-app:
    image: gcr.io/google-samples/hello-app:2.0
    environment:
      - PORT=8080
    labels:
      traefik.enable: true
      traefik.http.routers.hello-app.rule: Host(`hello-app.docker.localhost`)
      traefik.http.routers.hello-app.entrypoints: http
      traefik.http.routers.hello-app.service: hello-app

Step 3: Run the Services and Generate Traffic

Start the services using Docker Compose:

docker compose up -d

Once the services are running, generate some traffic to the hello-app service. This will create traces and metrics that Traefik will send to SigNoz.

curl -H Host:hello-app.docker.localhost http://127.0.0.1

Step 4: Verify in SigNoz

1. Open your SigNoz UI.
2. Go to the Traces tab to see the traces generated by your requests.
3. Go to the Metrics Explorer tab and search for traefik to see the incoming metrics.
4. Go to the Dashboards tab to visualize metrics. You can create a new dashboard and add widgets to plot Traefik metrics. Learn more about creating dashboards here.

You can also check the list of available metrics below to know what you can visualize.

List of Metrics

Traefik Metrics

• traefik_config_last_reload_success
• traefik_config_reloads_total
• traefik_entrypoint_request_duration_seconds_bucket
• traefik_entrypoint_request_duration_seconds_count
• traefik_entrypoint_request_duration_seconds_sum
• traefik_entrypoint_requests_bytes_total
• traefik_entrypoint_requests_total
• traefik_entrypoint_responses_bytes_total
• traefik_open_connections
• traefik_service_request_duration_seconds_bucket
• traefik_service_request_duration_seconds_count
• traefik_service_request_duration_seconds_sum
• traefik_service_requests_bytes_total
• traefik_service_requests_total
• traefik_service_responses_bytes_total