Reduce Trace Volume with Sampling Strategies

This guide covers the primary approaches to reducing trace ingestion volume in SigNoz.

Trace volume is measured in Gigabytes (GB) ingested. High-frequency, low-latency successful requests (health checks, background polling, routine CRUD operations) generate disproportionate span volume relative to their diagnostic utility.

Identify which span operations are generating the highest volume, assess whether they have operational value, and apply the appropriate solution below.

APM metrics (RED metrics such as request rate, error rate, and latency) are computed from ingested trace data and are not billed separately.

Solution 1: Head-Based Sampling

Head-based sampling makes the keep/drop decision at trace creation time in the SDK, before any span data is serialized or transmitted. It requires no Collector memory and no gateway Collector.

Trade-off: The sampling decision is made before the trace completes, so error traces have the same drop probability as successful ones. You cannot guarantee 100% error trace retention. If that is a requirement, use tail-based sampling instead.

Configure the ParentBased(TraceIdRatioBased) sampler via environment variable:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1  # Retain 10% of all traces

parentbased_traceidratio is the correct sampler for distributed systems. It respects the parent span's sampling decision so the entire trace is kept or dropped together. Plain traceidratio ignores the parent's sampled flag: it still applies a deterministic hash of the trace ID, but each service evaluates that hash independently. If every service in the trace uses the same ratio, the decisions happen to agree; as soon as services run different ratios, the trace ID hash can pass in one service and fail in another, producing broken partial traces.

Sampling strategies and trade-offs: covers ParentBased(TraceIdRatioBased), reservoir sampling, adaptive sampling, and byte-rate limiting.

Solution 2: Tail-Based Sampling

Tail sampling buffers the complete trace in Collector memory before making a sampling decision. This lets you retain 100% of error traces and slow traces while dropping healthy traffic, a guarantee head-based sampling cannot provide.

A typical cost-saving policy set keeps every error trace, keeps every slow trace above a latency threshold, and applies a low-percentage sample to everything else. Policies are OR-ed: a trace is kept if it matches any policy. Add this to your existing otel-collector-config.yaml and enable tail_sampling in the traces pipeline:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow-traces
        type: latency
        latency:
          threshold_ms: 2000
      - name: sample-the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 3

Adjust threshold_ms and sampling_percentage to your own latency SLO and cost target. For full setup instructions, policy reference, memory and topology requirements, and APM metrics impact, see Tail Sampling.

Solution 3: Pre-Filter Before Sampling

Dropping a span at the Collector does not automatically remove custom metrics generated from that span via the spanmetrics processor. Metric volume is managed independently. See Optimize Metrics.

Two common drop patterns based on what you found in Trace Explorer:

By span operation name (the name field surfaced in Step 4 of the analysis guide):

processors:
  filter/drop_noise:
    error_mode: ignore
    traces:
      span:
        - 'name == "grpc.health.v1.Health/Check"'
        - 'IsMatch(name, "GET /internal/.*")'

The filter processor evaluates each span independently. If a matched span is a parent with child spans, dropping it orphans its children instead of removing the whole subtree, leaving a broken trace. Scope these patterns to leaf-level operations (like health check endpoints) that you've confirmed have no children.

By URL path attribute (for HTTP spans where the operation name alone is too broad):

processors:
  filter/drop_noise:
    error_mode: ignore
    traces:
      span:
        - 'IsMatch(attributes["url.path"], ".*health.*")'
        # Adjust the pattern to match your actual endpoint paths before deploying

Both patterns can be combined in a single filter/drop_noise processor block. Pipeline ordering is critical: filter/drop_noise must execute before tail_sampling:

service:
  pipelines:
    traces:
      processors: [filter/drop_noise, tail_sampling]

Validate Your Changes

The Cost Meter Dashboard uses signoz.meter.* metrics which aggregate hourly. Changes may take up to one hour to appear there. Use the Trace Explorer for faster feedback.

To confirm the volume reduction:

Navigate to Traces, swap to the Table tab. Filter by your target service, group by name (span operation name), and sort by count. Verify the high-volume operations you targeted have reduced.
Check the Cost Meter Dashboard after an hour, filtered by your target service, to confirm the span GB trend has dropped.

Next Steps

Configure Cost Meter Alert to be notified when trace volume crosses your expected threshold.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.