Complete Guide to systemctl Logs: View, Filter, and Troubleshoot Linux System Logs

When a Linux service fails, restarts unexpectedly, or behaves differently after a deployment, many admins start with the systemctl status command as it shows the current service state and a few recent log lines, commonly phrased as systemctl logs.

This guide explains what systemctl logs are, how systemctl status and journalctl can be used together to inspect logs for a specific service, and filter journal entries by time, priority, boot session, and other fields. It also covers journal storage, cleanup, and forwarding systemd logs to tools for centralized logging.

Understanding Systemctl Logs and the Systemd Ecosystem

What are Systemctl Logs?

systemctl is the primary interface for managing systemd, services and units. You use it to start, stop, restart, enable, disable, and inspect services. When you run the systemctl status command, it outputs the service state, process details, unit file information, and recent log entries from the journal. These log entries are generally referred to as systemctl logs.

How Systemd Logging Works

The systemd logging ecosystem consists of three key components:

1. systemd: The system and service manager
2. systemd-journald: The logging daemon that collects and stores logs
3. journalctl: The command-line tool for querying and analyzing logs

The systemd manages the system and services in Linux. The systemd-journald collects logs from several sources and stores them in a structured binary journal. These sources include service stdout and stderr streams, kernel messages, syslog-compatible messages, audit events, and logs from applications configured to write to the journal. Persistent logs are usually stored under /var/log/journal/ and volatile logs are usually stored under /run/log/journal/ paths.

The exact location depends on your journald configuration. Each log entry includes metadata such as timestamp, unit name, process ID, user ID, priority, hostname, and boot ID, which can be queried later by using journalctl.

Systemctl Logs vs Traditional Linux Service Logs

Before systemd, Linux service logs were usually spread across files under /var/log/, such as /var/log/syslog, /var/log/messages, /var/log/auth.log, or application-specific log files. That model still exists on many systems, but it can be harder to correlate events across services.

Systemd logs improve this in three ways:

For day-to-day troubleshooting, start with systemctl status and then move to journalctl when you need more control. journalctl lets you filter by service, time range, priority, boot session, and journal metadata, which is much harder to do with scattered plain-text files.

Basic Systemctl Commands for Service Monitoring

You can start with the following basic systemctl commands to check your service status or find why it failed.

Check service status and recent logs

Use this command first when a service fails. It gives you the current state and systemctl logs before you open the full journal.

sudo systemctl status <service-name>

Example:

sudo systemctl status nginx

This command shows:

• Current service state (active, inactive, failed)
• Process information and PID
• Recent log entries (last 10 lines by default)
• Service unit file location

Manage service lifecycle

# Start a service
sudo systemctl start <service-name>

# Stop a service
sudo systemctl stop <service-name>

# Restart a service
sudo systemctl restart <service-name>

# Reload configuration without a full restart, if supported
sudo systemctl reload <service-name>

# Enable service at boot
sudo systemctl enable <service-name>

# Disable service at boot
sudo systemctl disable <service-name>

List and inspect services

# List active services
sudo systemctl list-units --type=service

# List active and inactive services
sudo systemctl list-units --type=service --all

# List failed units
sudo systemctl --failed

# Show service dependencies
sudo systemctl list-dependencies <service-name>

# Show the unit file and drop-in overrides
sudo systemctl cat <service-name>

Understanding Systemctl Status Output

A typical systemctl status nginx output looks like this:

● nginx.service - A high-performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
     Active: active (running) since Mon 2026-05-11 10:30:15 UTC; 2h ago
   Main PID: 1236 (nginx)
      Tasks: 3
     Memory: 12.4M
        CPU: 1.234s
     CGroup: /system.slice/nginx.service
             ├─1236 nginx: master process /usr/sbin/nginx
             └─1237 nginx: worker process

May 11 10:30:15 server systemd[1]: Starting nginx.service...
May 11 10:30:15 server nginx[1234]: nginx: configuration file /etc/nginx/nginx.conf syntax is ok
May 11 10:30:15 server systemd[1]: Started nginx.service.

The most important fields are:

• Loaded: Shows whether the unit file was found and whether the service is enabled.
• Active: Shows the current service state.
• Main PID: Shows the main process managed by systemd.
• CGroup: Shows the process tree for the service.
• Recent log lines: Shows the latest systemd service logs for that unit.

If the service failed, check the exit code and the recent log lines at the bottom of the output. Then run journalctl -u <service-name> to see what happened before and after those lines.

Viewing Full Systemd Service Logs with journalctl

You can use the following basic journalctl commands to view the full journal or to filter logs by service, time, priority, boot session, or metadata field.

View all systemd logs

# Show all accessible journal entries
sudo journalctl

# Follow logs in real time
sudo journalctl -f

# Show newest entries first
sudo journalctl -r

View logs for a specific service

# View logs for one service
sudo journalctl -u nginx

# Follow logs for one service
sudo journalctl -u nginx -f

# View logs for multiple services
sudo journalctl -u nginx -u mysql -u redis

Filter Systemctl Logs by Time

Time-based filters are useful for incidents with a known window. Instead of reading the full journal, you can narrow the output to the period where the issue happened.

# Logs from the last hour
sudo journalctl --since "1 hour ago"

# Logs from today
sudo journalctl --since today

# Logs from yesterday
sudo journalctl --since yesterday

# Logs from a specific time range
sudo journalctl --since "2026-05-11 10:00:00" --until "2026-05-11 11:00:00"

Combine time filters with a service filter:

sudo journalctl -u nginx --since "30 minutes ago"

Filter Logs by Priority

Journal entries use syslog-style priority levels. During an outage, start with warnings or errors so you don't have to scan through routine informational messages.

# Show errors and more severe messages
sudo journalctl -p err

# Show warnings and more severe messages
sudo journalctl -p warning

# Show nginx errors only
sudo journalctl -u nginx -p err

# Show errors from the last hour
sudo journalctl -u nginx --since "1 hour ago" -p err

Priority levels in order of severity::

Filter Logs by Boot Session, PID, or Journal field

systemd journal contains structured fields with each entry, so you can filter beyond service name and time.

Boot-based filtering

# Logs from the current boot
sudo journalctl -b

# Logs from the previous boot
sudo journalctl -b -1

# List recorded boots
sudo journalctl --list-boots

For boot failures, combine boot and priority filters:

sudo journalctl -b -1 -p err

Process and user filters

# Logs for a specific PID
sudo journalctl _PID=1234

# Logs for a specific user ID
sudo journalctl _UID=1000

# Logs for a specific executable
sudo journalctl /usr/sbin/nginx

Unit field filter

sudo journalctl _SYSTEMD_UNIT=nginx.service --since today

Troubleshooting a Failed Service

When a service fails, use the following workflow:

1. Check the service state

sudo systemctl status <service-name>

Look for the active state, exit code, recent errors, and whether the service restarted repeatedly.

2. Read detailed service logs

sudo journalctl -u <service-name> --no-pager

For recent failures:

sudo journalctl -u <service-name> --since "1 hour ago" -p err

3. Check the unit file

sudo systemctl cat <service-name>

4. Check dependencies

sudo systemctl list-dependencies <service-name>

For checking reverse dependencies:

sudo systemctl list-dependencies <service-name> --reverse

5. Validate the unit file

sudo systemd-analyze verify /etc/systemd/system/<service-name>.service

Use this when you suspect a syntax issue in a custom unit file.

Troubleshooting Boot Time Issues

Boot problems usually involve multiple services, so start with boot-level logs and then narrow down.

# Current boot logs
sudo journalctl -b

# Current boot errors
sudo journalctl -b -p err

# Previous boot logs
sudo journalctl -b -1

# Previous boot errors
sudo journalctl -b -1 -p err

To inspect failed units:

sudo systemctl --failed

To analyze boot performance:

# Overall boot timing
systemd-analyze

# Services ordered by startup time
systemd-analyze blame

# Longest dependency chain
systemd-analyze critical-chain

Common Systemctl Log Messages and What They Mean

Started <service-name>.service

The service started successfully. No action is needed unless the start was unexpected.

Stopped <service-name>.service

The service stopped cleanly. If you did not expect it to stop, check the surrounding journal entries to see if it was not planned.

Failed to start <service-name>.service

The service startup failed. Check detailed logs:

sudo journalctl -u <service-name> --no-pager
sudo systemctl status <service-name> -l

Main Process Exited, code=exited, status=1/FAILURE

The service process exited with a non-zero status code. This usually points to an application error, a bad configuration, a missing dependency, or a failed startup check.

sudo journalctl -u <service-name> -p err

Main process exited, code=killed, status=9/KILL

The process was killed. This can happen after a manual kill, an OOM kill, or after systemd terminates the process due to a timeout. Check kernel logs:

sudo journalctl -k | grep -i "killed process"
dmesg | grep -i "out of memory"

Start operation timed out. Terminating.

The service did not start before its configured timeout. Check whether it is waiting on a dependency, running a slow migration, trying to reach another service, or doing heavy startup work.

sudo systemctl edit <service-name>

Then add an override only if a longer startup time is expected:

[Service]
TimeoutStartSec=300

Dependency failed for <service-name>.service

A required service or unit failed. Inspect dependencies:

sudo systemctl list-dependencies <service-name>
sudo systemctl status <dependency-name>

Failed to parse service file

The unit file has a syntax problem. Validate it:

sudo systemd-analyze verify /etc/systemd/system/<service-name>.service

Permission denied

The service cannot execute a file, read a path, bind to a port, or access a resource.

ls -la /path/to/executable
sudo journalctl -u <service-name> --since "30 minutes ago"

On systems using SELinux or AppArmor, also check the relevant security logs.

Managing Journal Storage and Rotation

If systemd logs grow without limits, they can consume disk space and slow down log queries. Configure retention in:

/etc/systemd/journald.conf

Example configuration:

[Journal]
Storage=persistent
Compress=yes
Seal=yes
SystemMaxUse=2G
SystemKeepFree=1G
SystemMaxFileSize=128M
MaxRetentionSec=1month
RateLimitIntervalSec=30s
RateLimitBurst=1000

After changing the configuration, restart journald:

sudo systemctl restart systemd-journald

Check journal disk usage:

sudo journalctl --disk-usage

Clean up old logs manually:

# Remove logs older than two weeks
sudo journalctl --vacuum-time=2weeks

# Limit journal size to 1 GB
sudo journalctl --vacuum-size=1G

# Keep only the last 10 journal files
sudo journalctl --vacuum-files=10

Verify journal integrity:

sudo journalctl --verify

Exporting and Processing Systemd Logs

For deeper analysis, export systemctl logs in structured formats.

# JSON output
sudo journalctl -u nginx -o json

# Pretty JSON output
sudo journalctl -u nginx -o json-pretty

# Export today's nginx logs
sudo journalctl -u nginx --since today -o json > nginx-logs.json

You can combine journalctl with jq:

sudo journalctl -u nginx -o json | jq -r '.MESSAGE'

For quick pattern searches, grep is still useful:

sudo journalctl -u webapp.service | grep -i "database\|connection\|timeout"

Use native journalctl filters first, where possible, then use grep or jq for message-level processing on systemctl logs.

Sending Systemd Service Logs to SigNoz

The journalctl workflow works well on a single server. Once the same service runs on multiple hosts, checking logs means SSH-ing into each machine and running the same query again. It also makes cross-host patterns easy to miss, such as the same service failing after a deployment or the same permission error appearing on several machines. Forwarding systemd logs to a centralized platform puts those entries in one place, so you can search across hosts instead of investigating them one by one.

SigNoz can collect Linux service logs from every server and make them searchable from a single interface. From there, you can filter by service or host, compare failures across machines, and correlate logs with metrics and traces from the same systems. This is especially useful for problems that appear as patterns rather than isolated log lines, such as restart loops, recurring permission errors, or a service failing across multiple hosts.

You can send systemd logs to SigNoz through the OpenTelemetry Collector with the journald receiver from the Collector Contrib distribution. The receiver reads from the systemd journal and maintains structured fields such as _SYSTEMD_UNIT and PRIORITY, making filtering easier after the logs are ingested. It also avoids treating journal data as ordinary text files, which can cause loss of useful systemd metadata. Here is a minimal configuration that sends systemd service logs to a SigNoz endpoint:

Example configuration:

receivers:
  journald:
    directory: /var/log/journal
    start_at: end

processors:
  batch:

exporters:
  otlp:
    endpoint: "<your-signoz-otlp-endpoint>:4317"
    headers:
      signoz-ingestion-key: "<your-ingestion-key>"

service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [batch]
      exporters: [otlp]

If /var/log/journal does not exist on your system, check whether the journal is stored under /run/log/journal or /run/journal.

Also, make sure the Collector can run journalctl and has permission to read the journal. On host-based Collector installations, this is usually simpler. In containers, you need to mount the journal directory and provide a compatible journalctl binary.

Getting started with SigNoz

SigNoz provides log management to search, filter, and analze logs across services and hosts. Once systemd service logs are flowing into SigNoz, you can correlate them with metrics and traces, rather than troubleshooting each host in isolation.

You can use SigNoz Cloud for a managed setup, enterprise self-hosted or BYOC for stricter data control requirements, or the community edition for a self-hosted deployment.

Best Practices for Working with Systemctl Logs

When managing systemctl logs in production environments, follow these best practices:

Security and Access Control

Protect sensitive information:

# Configure journald to limit log access
sudo chmod 640 /var/log/journal/*/*
sudo chown root:systemd-journal /var/log/journal/*/*

# Use systemd user sessions for application isolation
loginctl list-sessions
journalctl --user -u user-service

Implement log rotation policies:

# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=2G
SystemKeepFree=500M
SystemMaxFileSize=100M
MaxRetentionSec=1month
MaxFileSec=1week
Compress=yes
Seal=yes

Performance Optimization

For efficient systemd logs processing:

Optimize performance:

# Monitor journal performance impact
systemctl status systemd-journald
journalctl -u systemd-journald --since "1 hour ago"

# Configure rate limiting for verbose services
# In service unit file:
[Service]
StandardOutput=journal
StandardError=journal
SyslogLevel=info  # Reduce verbosity

Efficient log querying practices:

# Use specific time ranges to limit data processing
sudo journalctl -u service --since "1 hour ago" --until "30 minutes ago"

# Leverage binary format advantages
sudo journalctl -u service -o json-pretty | head -100

# Use field-specific queries when possible
sudo journalctl _SYSTEMD_UNIT=nginx.service --since today

Key takeaways

Start investigation with systemctl logs when a service fails, and if required, query the full log history in the systemd journal with journalctl.

For quick checks, run:

sudo systemctl status <service-name>

For full service logs, run:

sudo journalctl -u <service-name>

For focused troubleshooting, combine service, time, priority, boot, and field filters. For production environments, configure journal retention and centralize systemd logs so service failures are easier to detect and investigate across hosts.

FAQs

How do I see systemctl service logs?

Use sudo systemctl status <service-name> for recent logs and sudo journalctl -u <service-name> for the full systemd service logs for that unit.

Where are systemctl logs stored?

The logs are stored by systemd-journald, usually under /var/log/journal/ for persistent storage or /run/log/journal/ for volatile storage.

What is systemctl in Linux?

systemctl is the command-line tool used to inspect and control systemd. You can use it to start, stop, restart, enable, disable, and check the status of services and other systemd units.

How do I view system logs in Linux?

View logs using journalctl for systemd logs or access files in the /var/log/ directory using commands like cat, less, or tail.

Why is systemctl status slow?

Slow systemctl status output is often caused by a large journal, slow disk I/O, or a service producing too many logs. Check journal size with:

sudo journalctl --disk-usage

Then configure retention limits in /etc/systemd/journald.conf.

How do I clean up systemd journal logs?

Use one of the vacuum commands:

sudo journalctl --vacuum-time=2weeks
sudo journalctl --vacuum-size=1G
sudo journalctl --vacuum-files=10

These commands remove old archived journal data based on time, size, or file count.

Can I view logs from multiple services together?

Yes, use sudo journalctl -u service1 -u service2 -u service3 to view logs from multiple services in chronological order.

How do I analyze boot-time service failures?

Use sudo journalctl -b -p err for current boot errors, sudo journalctl -b -1 for previous boot logs, and systemd-analyze blame for startup timing analysis.