Collecting systemd logs (journald)

Overview

This documentation provides detailed instructions on configuring the OpenTelemetry Collector to read logs from systemd's journal (journald) and push them to SigNoz. Logs from systemd are structured and contain rich metadata that can help you monitor system services, troubleshoot issues, and track service performance.

Prerequisites

• Linux-based operating system
• The journalctl binary is available in your $PATH
• Systemd logs are available. Verify this by running:

  sudo journalctl -n 10
  

The systemd journal contains structured logs for all services managed by it. Typical journal entries look like this:

Jun 25 10:30:45 hostname systemd[1]: Started myapp.service.
Jun 25 10:30:46 hostname myapp[1234]: Application started successfully

Setup

Step 1: Install OpenTelemetry Collector Contrib

Follow the installation guide to install the OpenTelemetry Collector. The journald receiver is available in the OpenTelemetry Collector Contrib distribution.

Step 2: Configure the journald receiver

Add the journald receiver to your config.yaml file and enable it in the logs pipeline:

receivers:
  journald:
    directory: /var/log/journal
    start_at: end

If /var/log/journal doesn't exist on your host, try directory: /run/journal or directory: /run/log/journal (common default).

Then enable the receiver in your logs pipeline:

service:
  pipelines:
    logs:
      receivers: [otlp, journald]
      processors: [batch]
      exporters: [otlp]

If you don't already have an OTLP exporter configured for SigNoz Cloud, add the following snippet (or update your existing otlp exporter):

exporters:
  otlp:
    endpoint: ingest.<region>.signoz.cloud:443
    tls:
      insecure: false
    headers:
      signoz-ingestion-key: <SIGNOZ_INGESTION_KEY>

Replace the placeholders:

• <region>: Your SigNoz Cloud region (e.g., us, eu, or in)
• <SIGNOZ_INGESTION_KEY>: Your SigNoz ingestion key

Additional Configuration Options:

• start_at: end - Only collect new logs after the collector starts (default)
• start_at: beginning - Read all messages from the current boot
• units - Filter logs from specific systemd services
• priority - Filter by log level (debug, info, notice, warning, err, crit, alert, emerg)
• matches - Advanced filtering using journalctl match syntax
• storage - Track cursor position across restarts to avoid reading the entries again (see Cursor tracking)

See the journald receiver documentation for all available options.

Step 3: Start the OTel Collector

Start the OpenTelemetry Collector:

./otelcol-contrib --config ./config.yaml

Validate

After starting the collector, verify that systemd logs are being ingested:

1. Navigate to Logs Explorer in SigNoz from Sidebar
2. You should see logs with systemd-specific fields like _SYSTEMD_UNIT, PRIORITY, and _PID

If you don't see logs within a few minutes, check the Troubleshooting section.

Log fields from systemd

Logs from systemd include rich metadata. Common fields include:

• MESSAGE - The actual log message
• _SYSTEMD_UNIT - systemd unit that generated the log
• PRIORITY - Log priority (0-7, where 0 is emergency, 7 is debug)
• _PID - Process ID
• _HOSTNAME - System hostname
• SYSLOG_IDENTIFIER - Program name
• _COMM - Command name

receivers:
  journald:
    directory: /var/log/journal
    units:
      - "nginx.service"
      - "postgresql.service"
      - "myapp.service"

receivers:
  journald:
    directory: /var/log/journal
    priority: warning  # Only warning, err, crit, alert, emerg

receivers:
  journald:
    directory: /var/log/journal
    matches:
      - _TRANSPORT: kernel
      - _SYSTEMD_UNIT: ssh
        _UID: "1000"

( dmesg )
AND
( priority )
AND
( units[0] OR units[1] OR units[2] ... )
AND
( identifiers[0] OR identifiers[1] OR identifiers[2] ... )
AND
( matches[0] OR matches[1] OR matches[2] ... )
AND
( grep )

receivers:
  journald:
    directory: /var/log/journal
    priority: info
    units:
      - ssh
      - nginx
    matches:
      - _SYSTEMD_UNIT: ssh
      - _SYSTEMD_UNIT: nginx
        _UID: "0"

receivers:
  # Receiver for application logs - collect all logs including info and debug
  journald/app:
    directory: /var/log/journal
    priority: debug
    units:
      - myapp.service

  # Receiver for system services - only warning and above
  journald/system:
    directory: /var/log/journal
    priority: warning
    units:
      - nginx.service
      - postgresql.service

  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlp:
    endpoint: "ingest.<region>.signoz.cloud:443"
    tls:
      insecure: false
    headers:
      signoz-ingestion-key: <SIGNOZ_INGESTION_KEY>

service:
  pipelines:
    logs:
      receivers: [otlp, journald/app, journald/system]
      processors: [batch]
      exporters: [otlp]

# Check journal access
sudo journalctl --verify

sudo journalctl -n 10

Next Steps

Now that you're collecting systemd logs, explore these related features:

• Set up log pipelines to parse, transform, and enrich your logs
• Create alerts based on specific log patterns or error rates
• Build custom dashboards to visualize system service health
• Explore other log collection methods:
  • Collect application logs
  • Collect Docker container logs
  • Collect Kubernetes logs