Migrate logs from ELK Stack

Migrating logs involves replacing Filebeat and/or Logstash with the OpenTelemetry (OTel) Collector, configured to collect, process, and forward logs to SigNoz.

Replacing Log Shippers and Processors:

Several strategies can be employed, depending on the complexity of the existing ELK logging pipeline:

• Strategy 1: Replacing Filebeat with OTel agent:
  • This is the most direct approach for collecting logs from files, replacing Filebeat's role.
  • Deploy the OTel Collector as an agent(e.g., DaemonSet in Kubernetes , standalone binary on VMs ) where logs are generated.
  • Configure the logs in SigNoz to target the necessary log files.
  • Key configurations include:
    • include: Glob patterns for log file paths (e.g., /var/log/app/*.log).
    • exclude: Glob patterns to exclude specific files.
    • start_at: end (tail new logs) or beginning (read from start).
    • multiline: For logs spanning multiple lines (e.g., stack traces), using line_start_pattern or line_end_pattern.
    • operators: A pipeline of parsing and transformation steps applied within the receiver itself (e.g., regex_parser, json_parser, timestamp, severity, attribute).This can handle basic parsing previously done by Filebeat modules or simple Logstash filters.
• Strategy 2: Forwarding from Logstash to SigNoz(Transitional):
  • If you have complex Logstash pipelines you wish to retain temporarily, you can configure Logstash to forward logs to an SigNoz.
  • The OTel Collector then forwards the received logs to SigNoz.

Verification:

• Ensure logs are appearing in the SigNoz Logs Explorer.
• Verify that timestamps, severity levels, and attributes are parsed correctly according to the processor configurations.
• Check that filters are working as expected (e.g., debug logs are dropped).
• If trace context (trace IDs, span IDs) is expected in logs, confirm it's being parsed and that log-trace correlation works in the SigNoz UI.

Signoz logs features

• Log Pipelines: Parse and structure logs with processors like JSON, regex, and grok
• Log Querying: Advanced query capabilities for searching and filtering logs
• Log-Trace Correlation: Link logs with related traces for contextual debugging