Control OpenTelemetry Metric Cardinality with a Processor

Overview

One mislabeled metric can spawn thousands of new time series and overwhelm your metrics backend. Bake a user ID, a full request path, or a unique pod name into an attribute and every distinct value becomes its own series. This is a cardinality explosion.

This guide shows how to run the community Cardinality Guardian processor in your OpenTelemetry Collector. It flags labels whose value count is spiking and contains them before the collector exports the data to SigNoz. It strips only the offending label from a metric, not the whole data point, so the rest of your dashboards keep working.

A few things to know before you start:

• Metrics only. The processor operates on metric data. Logs and traces are unaffected.
• Naming quirk. The component built into the collector is named cardinalityprocessor, but the key you use in the processors: block of your config is cardinality_guardian.
• Where it runs. You add it to a collector that you run. SigNoz Cloud's managed collector cannot load custom processors, so this sits in your own collector, upstream of the OTLP export to SigNoz.

Prerequisites

• An OpenTelemetry Collector you control, exporting metrics to SigNoz.
• Go toolchain, to build a custom collector with the OpenTelemetry Collector Builder (OCB).
• An instance of SigNoz (either Cloud or Self-Hosted).

How it works

The processor tracks the number of distinct values for each metric attribute during a fixed time window. It compares that count with the previous window. If the increase exceeds your configured threshold, the processor flags the attribute and applies the selected enforcement mode.

Build a custom collector with OCB

The processor is not in the default SigNoz or contrib collector images, so you build a collector binary that includes it using the OpenTelemetry Collector Builder (OCB).

1. Download OCB matching your OS and the collector version you want to build. Check the collector-releases page for the current version and replace <version> and the platform suffix below:

   curl --proto '=https' --tlsv1.2 -fL -o ocb \
     https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv<version>/ocb_<version>_darwin_arm64
   chmod +x ocb
   

2. Create builder.yaml. Keep every gomod version aligned to the same collector release you downloaded OCB for. The cardinalityprocessor entry is what pulls in the community processor:

   builder.yaml

   dist:
     name: otelcol-custom
     description: Custom OTel Collector with the cardinality processor
     output_path: ./build
   
   receivers:
     - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v<version>
   
   processors:
     - gomod: go.opentelemetry.io/collector/processor/batchprocessor v<version>
     - gomod: github.com/YElayyat/otel-cardinality-processor v1.5.1
       name: cardinalityprocessor
       import: github.com/YElayyat/otel-cardinality-processor/cardinalityprocessor
   
   exporters:
     - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v<version>
   

   The custom binary only contains the components you list here. It must include every receiver, processor, and exporter your existing otel-collector-config.yaml references, or the collector fails to start with an "unknown type" error. The example above uses otlphttpexporter because the standard SigNoz Cloud collector config exports over OTLP/HTTP; add any others your config uses.

3. Compile:

   ./ocb --config=builder.yaml
   

   OCB writes the binary to ./build/otelcol-custom.

docker pull ghcr.io/yelayyat/otel-cardinality-processor:1.5.1

Configure the processor

Add the cardinality_guardian block to your existing otel-collector-config.yaml and reference it in your metrics pipeline. Do not replace the rest of your config.

processors:
  cardinality_guardian:
    max_cardinality_delta_per_epoch: 100
    epoch_duration_seconds: 300
    enforcement_mode: tag_only
    never_drop_labels:
      - service.name
      - deployment.environment
    metric_overrides:
      http.server.request.duration: 5000

Enable it in the pipeline. Place it after batch and before your exporter. The processor tracks unique attribute values per epoch, so batching does not affect detection, but running after batch means it operates on the final data shape the exporter sends. The telemetry block exposes the processor's own diagnostics on :8888, used in Validate:

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, cardinality_guardian]
      exporters: [otlphttp]

The values service.name and http.server.request.duration above are examples. Replace them with the attributes and metrics in your own telemetry.

Export diagnostics to SigNoz (optional)

Those :8888 diagnostics stay outside your pipelines. To send them to SigNoz, add a prometheus receiver that self-scrapes :8888 and a diagnostics pipeline that reuses your existing exporter. Add prometheusreceiver to your builder.yaml so the build includes it.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: cardinality-guardian
          static_configs:
            - targets: ['localhost:8888']

service:
  pipelines:
    metrics/diagnostics:
      receivers: [prometheus]
      exporters: [otlphttp]

Enforcement modes

enforcement_mode controls what happens to an attribute once it is flagged:

Validate

Start in tag_only mode so nothing is dropped while you confirm the processor works.

1. Find the offending label. In SigNoz Metrics Explorer, open the Summary tab and click the metric to see its time-series count and per-attribute cardinality.

2. Confirm the processor is acting. It publishes diagnostics on the collector's Prometheus endpoint, port 8888. Read them with curl localhost:8888/metrics, or use the optional diagnostics scraper to view them in SigNoz. The useful ones, as they appear in SigNoz:

• otelcol_processor_cardinality_labels.stripped_total: its rate spikes when enforcement fires.
• otelcol_processor_cardinality_top.offenders: names the highest-growth label, so you can spot the culprit.
• otelcol_processor_cardinality_trackers.active: how many pairs are currently tracked.

On the raw :8888 endpoint these render with underscores instead of the dot, for example otelcol_processor_cardinality_labels_stripped_total.

Next steps

• Dropping Metric Labels - why dropping labels alone doesn't reduce samples, and what to do instead.
• Understanding Metrics Billing and Reducing Costs - strategies to keep metric costs under control.
• Metrics Explorer - inspect series and cardinality for a metric.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.