Control Traces Volume

Overview

You can control span volume at two stages:

1. At the application — customize the SDK or instrumentation
2. At the collector — use processors in the OTel Collector

Prerequisites

• OTel Collector contrib distribution (otelcol-contrib) — the filter, attributes, resource, and sampling processors ship in contrib, not in the core distribution.
• Filter processor v0.146.0 or later for trace_conditions syntax. The older traces.span format still works but is deprecated and no longer documented upstream.

At the application

Dropping spans at the application is HEAD sampling. The sampling decision happens at the start of the trace.

TraceIdRatioBased sampler

The TraceIdRatioBased sampler drops spans by probability. At 0.001, 1 in 1000 traces is sampled. Set it with OTEL_TRACES_SAMPLER:

OTEL_TRACES_SAMPLER="parentbased_traceidratio"
OTEL_TRACES_SAMPLER_ARG=0.001

Custom sampler

The Sampler interface lets you write custom sampling logic. Drop spans by condition, not just by ratio. Your sampler must implement the ShouldSample method, which returns one of:

• DROP — span is dropped
• RECORD_ONLY — span is recorded but not exported
• RECORD_AND_SAMPLE — span is recorded and exported

See your SDK's documentation for implementation details.

Exclude certain routes

Some instrumentation libraries let you exclude routes from generating spans. OpenTelemetry Python uses OTEL_PYTHON_EXCLUDED_URLS:

OTEL_PYTHON_EXCLUDED_URLS="https://example.com/exclude"

See your instrumentation library's documentation for the equivalent option.

At the collector

Three approaches at the collector level:

1. Drop specific attributes from spans
2. Drop entire traces with the tail or probabilistic sampling processor
3. Drop individual spans with the filter processor

Sampling processors (tail, probabilistic) evaluate complete traces at the collector — this is TAIL sampling. The filter and attribute processors operate on individual spans as they arrive.

Drop certain attributes from spans

Two processors handle attribute removal:

• Attributes processor — removes span-level attributes
• Resource processor — removes resource-level attributes

Add the attributes processor to your traces pipeline in otel-collector-config.yaml, for example:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/drop_span_attributes, batch]
      exporters: [otlp]

Or for the resource attributes:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource/drop_span_resource_attributes, batch]
      exporters: [otlp]

Drop a span attribute (http.method):

processors:
  attributes/drop_span_attributes:
    actions:
      - key: http.method
        action: delete

Drop multiple resource attributes:

processors:
  resource/drop_span_resource_attributes:
    attributes:
      - key: process.runtime.description
        action: delete
      - key: telemetry.distro.name
        action: delete
      - key: process.executable.path
        action: delete
      - key: process.runtime.name
        action: delete
      - key: process.command_args
        action: delete

Drop entire traces

Two processors drop full traces:

1. Probabilistic Sampling Processor — drops traces at a configured ratio
2. Tail Sampling Processor — drops traces based on policies evaluated after all spans arrive

See the OTel documentation for probabilistic sampling and tail sampling.

Filter processor

The filter processor drops spans that match OTTL conditions. Add it to your otel-collector-config.yaml and enable it in the traces pipeline:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter, batch]
      exporters: [otlp]

Each condition in trace_conditions uses an OTTL context prefix (span., resource., spanevent., or scope.) to identify the field. A span is dropped when any condition matches (conditions are OR-ed).

Set error_mode to control behavior when a condition fails to evaluate:

Drop by span name

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.name == "health-check"

Drop by status

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.status.code == STATUS_CODE_OK

Drop by resource attribute

Drop all spans from a specific pod, service, or host:

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - resource.attributes["k8s.pod.name"] == "test-pod"

Drop by resource attribute (regex)

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - IsMatch(resource.attributes["k8s.pod.name"], "test-pod-.*")

Drop by span attribute

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.attributes["http.method"] == "GET"

Drop by span attribute (regex)

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - IsMatch(span.attributes["http.method"], "GET|POST")

Drop non-error spans under 1 second

Keep only slow spans and error spans. Discard the fast, successful ones:

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - (span.end_time - span.start_time) < Duration("1s") and span.status.code != STATUS_CODE_ERROR

Drop spans using multiple conditions

Conditions are OR-ed. A span matching any one of them is dropped:

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.attributes["container.name"] == "sidecar"
      - resource.attributes["host.name"] == "localhost"
      - span.name == "internal-probe"

Drop spans missing an attribute

Drop spans that have no http.request.method attribute (non-HTTP spans):

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.attributes["http.request.method"] == nil

Flip the condition to drop all HTTP spans instead:

processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.attributes["http.request.method"] != nil

See the OTel filter processor documentation for the full OTTL condition reference.

Validate

After applying the processor config and restarting the collector, confirm that spans are being dropped:

1. Open Traces > Explorer in SigNoz.
2. Compare the span count for the affected service before and after the config change.
3. Verify that spans matching your filter conditions no longer appear.

If the collector exposes Prometheus metrics, check the otelcol_processor_filter_spans_filtered counter to see how many spans the filter processor is dropping.

service:
  telemetry:
    logs:
      level: debug

Next Steps

• PII Scrubbing in Traces — remove sensitive attribute values from spans before they reach SigNoz
• APM Metrics — monitor request rate, error rate, and latency after controlling trace volume
• Correlate Traces and Logs — navigate between traces and logs once your trace data is clean

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack.

If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.