Logs Parsing with Pipelines

Log parsing transforms unstructured log messages into structured data, allowing you to filter, aggregate, and analyze your logs efficiently in SigNoz.

Why Parse Logs?

Consider this raw application log:

2025-12-27 10:00:00 [ERROR] User 456 failed login from 10.0.0.5 - invalid password

Without parsing, you can only search the full text. You cannot filter by log level, count errors per user, or identify suspicious IPs.

After parsing, the log becomes structured:

{
  "body": "2025-12-27 10:00:00 [ERROR] User 456 failed login from 10.0.0.5 - invalid password",
  "attributes": {
    "level": "ERROR",
    "user_id": "456",
    "ip": "10.0.0.5",
    "reason": "invalid password"
  }
}

With structured attributes, you can:

• Filter logs by level = ERROR or user_id = 456
• Aggregate failed logins by IP address to detect brute-force attempts
• Build dashboards showing error rates over time
• Create alerts when a specific user exceeds failed login thresholds

Prerequisites

• You are sending logs to SigNoz.
• You have access to the Logs Pipelines section in SigNoz UI.

Learn more about SigNoz Logs Pipelines.

Creating a Pipeline

Before you can parse logs, you need to create a pipeline that will filter and process them.

Step 1: Navigate to Pipelines Page

In SigNoz, go to Logs → Pipelines

Step 2: Create a New Pipeline

• If you do not have existing pipelines, press the "New Pipeline" button.

  

• If you already have some pipelines, press the "Enter Edit Mode" button and then click the "Add a New Pipeline" button at the bottom of the list.

Step 3: Configure the Pipeline

Provide details about the pipeline in the Create Pipeline Dialog:

• Name: Provide a descriptive pipeline name.
• Description: Add a detailed description for your pipeline (optional).
• Filter: Use the filter field to select the logs you want to process. For example, service.name = my-app or log.file.name = app.log.
• Filtered Logs Preview: Verify that the logs you want to process are selected.

Press the "Create" button to create the pipeline.

Choosing the Right Parser

SigNoz offers several processors to handle different types of logs. Choosing the right one depends on your log format.

JSON Parser

If your logs are formatted as JSON strings, use the JSON Parser. It automatically expands the JSON structure into log attributes.

Best for: Structured logs generated by modern applications.

Read the JSON Parsing Guide

Regex Parser

If your logs are unstructured text with a predictable pattern, use the Regex Parser. It allows you to define flexible named capture groups using regular expressions.

Best for: Custom or legacy application logs.

Read the Regex Parsing Guide

Grok Parser

If your logs follow common standard formats (like Apache, Nginx, or Syslog), use the Grok Parser. It provides pre-defined patterns that are easier to use and maintain than raw regex.

Best for: Web server access logs, system logs, and standard middleware logs.

Read the Grok Parsing Guide

Specialized Parsers

• Trace Parser: Extract trace IDs from log messages and map them to the official trace ID field.
• Timestamp Parser: Extract timestamps from log messages and map them to the official timestamp field.
• Severity Parser: Map log levels (INFO, ERROR, etc.) to the OpenTelemetry severity number.
• Add Parser: Add a new attribute to the log.
• Remove Parser: Remove an attribute from the log.
• Move Parser: Move an attribute to a new location in the log.
• Copy Parser: Copy an attribute to a new location in the log.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack.

If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.