Collecting SystemD Logs (JournalD)

Overview

This documentation provides detailed instructions on configuring the OpenTelemetry Collector to read logs from SystemD's journal (JournalD) and push them to SigNoz. SystemD logs are structured and contain rich metadata that can help you monitor system services, troubleshoot issues, and track service performance.

Prerequisites

Before proceeding, ensure that:

Linux-based operating system
journalctl binary is available in the $PATH

Sample SystemD Logs

SystemD generates structured logs for all services managed by it. Here's what typical journal entries look like:

Jun 25 10:30:45 hostname systemd[1]: Started myapp.service.
Jun 25 10:30:46 hostname myapp[1234]: Application started successfully
Jun 25 10:30:47 hostname myapp[1234]: Processing user request

These logs contain rich metadata including timestamps, hostnames, process IDs, service names, and priority levels.

You can check the SystemD logs using the following command:

sudo journalctl -n 10

Setup

Step 1: Install OpenTelemetry Collector Contrib

The JournalD receiver is available in the OpenTelemetry Collector Contrib distribution.

Follow this documentation to install the OpenTelemetry Collector.

Step 2: Configure JournalD Receiver

Modify the config.yaml file created during the installation of the OpenTelemetry Collector to include the journald receiver:

config.yaml

receivers:
  journald:
    directory: /var/log/journal
    start_at: end

  # Keep your existing receivers
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  otlp:
    endpoint: "ingest.<region>.signoz.cloud:443"
    tls:
      insecure: false
    headers:
      "signoz-ingestion-key": "<your-ingestion-key>"
service:
  pipelines:
    logs:
      receivers: [otlp, journald]        #Add the journald receiver to pipelines
      processors: [batch]
      exporters: [otlp]

Set the <region> to match your SigNoz Cloud region
Replace <your-ingestion-key> with your SigNoz ingestion key

✅ Info

You can also include the below configuration options for JournalD receiver:

Configuration Options:

start_at: end - Only collect new logs after the collector starts
start_at: beginning - Include historical logs from the journal
units - Filter logs from specific SystemD services
priority - Filter by log level (debug, info, notice, warning, err, crit, alert, emerg)
matches - Advanced filtering using journalctl match syntax

More options available here

Step 3: Start the OTel Collector

Start the OpenTelemetry Collector using the above config.yaml file:

./otelcol-contrib --config ./config.yaml

Visualizing Logs in SigNoz

The SystemD logs will be visible in the Logs tab of SigNoz.

SystemD logs visible in SigNoz — *SystemD journal logs in SigNoz Logs Explorer*

Understanding SystemD Log Fields

SystemD logs include rich metadata. Common fields include:

MESSAGE - The actual log message
_SYSTEMD_UNIT - SystemD unit that generated the log
PRIORITY - Log priority (0-7, where 0 is emergency, 7 is debug)
_PID - Process ID
_HOSTNAME - System hostname
_TIMESTAMP - Timestamp when the log was generated
SYSLOG_IDENTIFIER - Program name
_COMM - Command name

Advanced Configuration

Filtering Logs

You can filter SystemD logs in several ways:

By SystemD Units

receivers:
  journald:
    directory: /var/log/journal
    units:
      - "nginx.service"
      - "postgresql.service"
      - "myapp.service"

By Priority Level

receivers:
  journald:
    directory: /var/log/journal
    priority: warning  # Only warning, err, crit, alert, emerg

By Custom Matches

receivers:
  journald:
    directory: /var/log/journal
    matches:
      - "_TRANSPORT=kernel"
      - "PRIORITY=6"

Troubleshooting

Permission Issues

If you see permission errors:

# Check journal access
sudo journalctl --verify

Log Generation

Check if SystemD is generating logs:

sudo journalctl -n 10