Reduce Log Ingestion Volume in SigNoz

This guide covers how to reduce log ingestion volume in SigNoz by filtering low-value log records before they reach ingestion.

Log ingestion volume is measured in Gigabytes (GB) and is often dominated by verbose DEBUG output and repetitive infrastructure probes.

If your application's logging framework supports level configuration, setting the appropriate log level at source is more effective than Collector-level filtering: it prevents the log from being generated at all. If you do not have control over the source, use the Collector-level approach below.

Filter Logs at the Edge

Use the filter processor in the OpenTelemetry Collector to drop data before it reaches SigNoz. Read Filtering and Dropping Logs for configuration patterns by severity and resource attributes.

Decision Guide

The log types below use severity_text values as examples.

Truncate Oversized Log Bodies

Full JSON payloads, stack traces, and verbose SQL statements are often worth keeping but larger than they need to be. See Limit Log Body Size for the transform processor configuration that caps log body size at the Collector.

Deduplicate Repetitive Logs

Health check retries, connection errors, and heartbeat messages often emit the same log line hundreds of times within a short window. Use the logdedup processor to collapse these into a single entry instead of dropping them. It aggregates identical log records (matched on body, resource attributes, severity, and log attributes) over a configurable interval, and emits one consolidated entry carrying log_count, first_observed_timestamp, and last_observed_timestamp. Add it to your existing otel-collector-config.yaml:

processors:
  logdedup:
    interval: 10s
    log_count_attribute: log_count
    exclude_fields:
      - attributes.request_id

Then enable it in the logs pipeline:

service:
  pipelines:
    logs:
      processors: [logdedup, batch]

Scope deduplication with conditions (OTTL expressions) instead of applying it to the whole pipeline, for example to only DEBUG/INFO severities:

processors:
  logdedup:
    interval: 10s
    conditions:
      - severity_number < SEVERITY_NUMBER_WARN

Validate Your Changes

To confirm the volume reduction:

1. Navigate to Logs, swap to the Table tab. Filter by your target service and group by severity_text. Verify the log counts for the severities you targeted have reduced.
2. Wait 1 hour, then check the Cost Meter Dashboard filtered by your target service and confirm the log GB trend has dropped.

Next Steps

• Configure a Cost Meter Alert on signoz.meter.log.size to be notified when log volume crosses your expected threshold.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.