Overview
The OpenTelemetry (OTel) Collector is a vendor-agnostic component that sits between your applications and SigNoz. While you can directly send telemetry data from your applications to SigNoz, using the OTel Collector provides significant advantages in production environments.
Key Benefits
1. Data Processing and Transformation
The OTel Collector acts as a powerful data processing layer between your applications and SigNoz, giving you fine-grained control over your telemetry data before it is exported to SigNoz.
Filtering
The Collector enables you to filter out noisy or low-value telemetry data significantly reducing costs and storage requirements. The filter processor has a lot of conditional logic which allows you to apply filters based on specific attributes, resource information, or custom attributes.
This is helpful in removing unnecessary traffic like health check endpoints, and internal monitoring requests.
Dropping Data
Sometimes you need to remove specific telemetry data entirely. The Collector allows you to drop individual spans or metrics based on their attributes, metrics names, etc. It comes in handy when your telemetry data has high cardinality attributes which can impact performance as well as storage costs.
You can check guides for dropping telemetry data in logs, metrics and traces.
PII Scrubbing
It is important to protect PII data for both security as well as compliance purposes. The Collector can automatically detect and remove Personally Identifiable Information (PII) before telemetry data leaves your infrastructure. This capability ensures compliance with regulations like GDPR and HIPAA without requiring any changes to your codebase.
You can configure the Collector to redact attributes which include email addresses, IP addresses, authentication tokens and other sensitive data using pattern matching with regex with the transform processor. This pre-process approach means the sensitive information never reaches SigNoz or long-term storage.
For more examples, you can check guides for scrubbing PII in logs and traces.
Data Transformation
Standardizing telemetry across different environments becomes simpler with the Collector's transformation capabilities. You can normalize attributes names and their values to make consistent telemetry data across different services and teams with the transform processor. The Collector also supports converting between different data formats, aggregating metrics before export, and enriching telemetry data with additional contextual information.
Check out complete list of processors from OpenTelemetry Registry.
2. Auto-Add Resource Attributes
The Collector automatically enhances your telemetry data with additional context information about the infrastructure and environment where your application runs.
Without any manual configuration, the Collector adds metadata including hostname, operating system, and other details about the environment. In cloud environments it automatically detects and adds provider specific metadata such as AWS instance IDs, GCP project information, Azure resource details, along with region and availability zone data.
For containerized applications, especially in Kubernetes, the Collector adds metadata with pod names, namespaces and container IDs.
Check out these guides for setting resource attributes using resource processor and resource detection processor.
3. Export to Multiple Destinations
The OTel Collector supports exporting to multiple destinations simultaneously, which is very helpful during migrations. This capability allows you to gradually migrate from existing observability tools to SigNoz.
This also allows you to configure long term storage using S3 exporter for example, you can check more about long term storage in Infinite Retention of OpenTelemetry Data doc.
4. Multiple Receiver Support
The OTel Collector can accept telemetry data from a wide variety of sources and in different formats like Jaeger, Prometheus, or Zipkin, in addition to OpenTelemetry-native OTLP, the Collector can ingest it all. This ability to support multiple receivers makes it an ideal central ingestion point for all your telemetry data, regardless of its origin.
The complete list of 50+ receivers can be found at: OpenTelemetry Registry.
For a detailed guide on configuring different receivers, see the Receivers section in the configuration guide.
5. Efficient Data Handling with Batching
The Collector's batch processor is a key feature for improving data ingestion efficiency. It groups telemetry data into batches before sending it to the backend, which significantly reduces network overhead and increases throughput.
Learn how to configure it in the Batch Processor section of the configuration guide.
6. Application Performance and Resource Offloading
The Collector offloads telemetry processing from your applications and services, significantly reducing resource consumption and performance impact. The applications don't need to handle batching, retries, encryption, or complex export logic, the Collector handles these tasks with ease.
7. Resilience and Reliability
The Collector provides built-in options to prevent data loss during network issues or backend outages. It has automatic retries, persistent storage, in-memory buffer features which safeguards telemetry data and makes sure network outages are handled properly.
8. Centralized Configuration and Management
Managing telemetry configuration directly in the collection instead of managing separate applications simplifies operations. This allows updating filter rules, adding new exporters, or changing attributes in telemetry data possible without redeploying applications.
Next Steps
To get started with the OpenTelemetry Collector and send data to SigNoz:
- Install the OpenTelemetry Collector: Follow our guide to get started with collection agents.
- Configure the Collector: Learn how to configure the OpenTelemetry Collector to process and route your telemetry data.
- Instrument your application and send data: Find instructions for your language or framework in our instrumentation guides to send telemetry data to the collector.
Get Help
If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack.
If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.
