Collecting Docker Container Logs

You can easily collect all your Docker container logs using OpenTelemetry Collector and perform advanced queries to analyze them efficiently using SigNoz.

Prerequisites

• Docker Engine 20.10+ installed and running
• Administrative access to the Docker host

Setup

Step 1: Install OpenTelemetry Collector

Install the OpenTelemetry Collector by following this document. If you have already installed the OpenTelemetry Collector, skip this step.

Step 2: Update Collector Configuration

Add the filelog receiver to your existing otel-config.yaml file to collect Docker container logs:

receivers:  
  # Add this receiver for Docker logs
  filelog:
    include: [/var/lib/docker/containers/*/*-json.log]
    poll_interval: 200ms
    start_at: end
    include_file_name: false
    include_file_path: false
    operators:
      - id: container-parser
        type: container
        format: docker

service:
  pipelines:
    # Add receiver to logs pipeline
    logs:
      receivers: [otlp, filelog]
      processors: [batch]
      exporters: [otlp]

Step 3: Update Docker Run Command

Restart the OpenTelemetry Collector with additional volume mounts to access Docker logs:

Stop existing collector

docker stop opentelemetry-collector
docker rm opentelemetry-collector

Run with Docker logs access

docker run --name opentelemetry-collector \
  --restart unless-stopped \
  --detach \
  --user 0:0 \
  --pid host \
  --network host \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc:/hostfs/proc:ro \
  -v /sys:/hostfs/sys:ro \
  -v /:/hostfs:ro \
  -v "$(pwd)/otel-config.yaml":/etc/otelcol-contrib/config.yaml \
  -e HOST_PROC=/hostfs/proc \
  -e HOST_SYS=/hostfs/sys \
  -e HOST_ROOT=/hostfs \
  otel/opentelemetry-collector-contrib:0.130.1

Step 4: Verify Log Collection

Check that the collector is processing Docker logs:

Check collector logs

docker logs opentelemetry-collector

Generate test logs

docker run --rm alpine sh -c 'for i in $(seq 1 10); do echo "Test log entry $i"; sleep 1; done'

View Logs in SigNoz

The Docker container logs will be visible in the Logs tab of SigNoz.

Troubleshooting

No Logs Appearing

1. Check file permissions:

   ls -la /var/lib/docker/containers/
   

2. Verify collector has access:

   docker exec opentelemetry-collector ls /var/lib/docker/containers/
   

3. Check collector logs for errors:

   docker logs opentelemetry-collector | grep -i error
   

Last updated: December 17, 2024