OpenTelemetry Collector - architecture and configuration guide

OpenTelemetry Collector is a stand-alone service provided by OpenTelemetry. It can be used as a telemetry-processing system with a lot of flexible configurations to collect and manage telemetry data. Let's do a deep dive on OpenTelemetry Collectors to understand how it works.

The first step in setting up observability with OpenTelemetry is instrumentation. The application code is instrumented with OpenTelemetry client libraries that help generate telemetry data like logs, metrics, and traces.

Once the telemetry data is generated, it can be exported directly to an observability backend or an OpenTelemetry Collector. The collector provides a vendor-neutral way to collect, process, and export your telemetry data(logs, metrics, and traces), and that’s why it is preferable to use a collector. The biggest advantage of using OpenTelemetry collectors is the flexibility to create different data pipelines.

OpenTelemetry collectors can be deployed in different ways. It can be deployed on each host machine as an agent. When the collector is deployed on host machines, you can directly collect host metrics like CPU usage, RAM, disk I/O metrics, etc.

You can also run your OpenTelemetry Collector as a stand-alone service. The client libraries of OpenTelemetry have an exporter that can be configured to send the telemetry data to the collector. Usually, a mixed pattern of OpenTelemetry Collectors is recommended to handle scale.

Before deep-diving into OpenTelemetry collectors, let's take a short detour to understand what OpenTelemetry is.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize the generation, collection, and management of telemetry data(Logs, metrics, and traces). It is incubated under Cloud Native Computing Foundation(Cloud Native Computing Foundation), the same foundation which incubated Kubernetes.

OpenTelemetry follows a specification-driven development and provides client libraries to instrument applications in most programming languages. Once you have instrumented with OpenTelemetry, you should be able to collect various telemetry signals like logs, metrics, and traces from it. And that's where OpenTelemetry Collector comes into the picture.

Why to use OpenTelemetry Collector?

An OpenTelemetry collector has three main functions - collect, process, and export the telemetry data collected. Let us first understand why it is a critical component of your observability architecture.

OpenTelemetry Collector gives you the flexibility to handle multiple data formats and offloads responsibility from the application to manage telemetry data.

List of reasons why to use OpenTelemetry Collector:

• It provides a vendor-agnostic way to collect telemetry data.
• The Collector offloads responsibility from the application to manage telemetry data, thereby reducing overhead and providing separate concerns from the application for any telemetry configuration.
• Using OpenTelemetry Collector, you can export telemetry data in multiple formats to multiple observability vendors of your choice.
• It enables quick config-based updates to your telemetry data pipeline. It is trivial to update a config file to receive data in another format.
• It can also help collect host metrics such as RAM, CPU, and storage capacity.

Architecture of OpenTelemetry collector

OpenTelemetry collector consists of three main components: Receivers, processors, and exporters.

Receivers

Receivers are used to get data into the collector. Currently, the collector supports over forty different types of receivers. You can use the receivers to configure ports and formats the collector can take data in. It could be push or pull-based.

You can receive data in multiple formats. It has a default OTLP format, but you can receive data in other popular open-source formats like Jaeger or Prometheus.

Processors

Processors are used to do any processing required on the collected data, like data massaging, data manipulation, or any change in the data as it flows through the collector. It can also be used to remove PII data from the collected telemetry data, which can be very useful.

You can also do things like batching the data before sending it out, retrying in case the exporting fails, adding metadata, tail-based sampling, etc.

Exporters

Exporters are used to export data to an observability backend like SigNoz. You can send out data in multiple data formats. You can send different telemetry signals to different backend analysis tools. For example, you can send traces to Jaeger and metrics to Prometheus.

With the combination of these three components, OpenTelemetry Collector can be used to build data pipelines. Pipelines are configured via a YAML configuration file easily. This provides flexibility to teams managing their telemetry data.

How to configure a OpenTelemetry collector?

You need to configure the three components of the OpenTelemetry collector described above. Once configured, these components must be enabled via pipelines within the service section. SigNoz comes with an OpenTelemetry collector installed. You can find the configuration file of SigNoz OpenTelemetry collector here.

Configuring Receivers

In the sample code shown below, we have two receivers:

1. OTLP
   Default OpenTelemetry protocol to transfer telemetry data. SigNoz receives telemetry data in OTLP format.

2. Jaeger
   You can also receive traces data in Jaeger format, which is a popular distributed tracing tool.

receivers:
      otlp:
        protocols:
          grpc:
          http:
      jaeger:
        protocols:
          grpc:
          thrift_http:

Configuring Processors

There are three processors in the code sample shown below:

1. Batch
   Batching helps better compress the data and reduce the number of outgoing connections required to transmit the data. This processor supports both size and time-based batching.

2. Memory limiter
   The memory limiter processor is used to prevent out-of-memory situations on the collector. Given that the amount and type of data a collector processes are environment-specific and resource utilization of the collector is also dependent on the configured processors, it is important to put checks in place regarding memory usage.

3. Queued retry
   This processor is highly recommended to configure for every collector as it minimizes the likelihood of data being dropped due to delays in processing or issues exporting the data.

processors:
      batch:
        send_batch_size: 1000
        timeout: 10s
      memory_limiter:
        # Same as --mem-ballast-size-mib CLI argument
        ballast_size_mib: 683
        # 80% of maximum memory up to 2G
        limit_mib: 1500
        # 25% of limit up to 2G
        spike_limit_mib: 512
        check_interval: 5s
      queued_retry:
        num_workers: 4
        queue_size: 100
        retry_on_failure: true

You can find detailed information about these processors and more in OpenTelemetry Collector GitHub repo.

Configuring Exporters

In this sample code, we have created two exporters.

1. kafka/traces
   This forwards collected traces to write to a kafka topic named as otlp_spans.

2. kafka/metrics
   This forwards collected metrics to write to a kafka topic named as otlp_metrics.

exporters:
      kafka/traces:
        brokers:
          - signoz-kafka:9092
        topic: 'otlp_spans'
        protocol_version: 2.0.0
      kafka/metrics:
        brokers:
          - signoz-kafka:9092
        topic: 'otlp_metrics'
        protocol_version: 2.0.0

You can also configure extensions that enable things like monitoring the health of OpenTelemetry Collector.

Extensions

Extensions provide capabilities on top of the primary functionality of the OpenTelemetry Collector.

In this example, we have enabled two extensions.

1. Health Check
   It enables a URL that can be used to check the status of the OpenTelemetry Collector.

2. Zpages
   It enables an HTTP endpoint that provides live data for debugging different components of the OpenTelemetry Collector.

extensions:
      health_check: {}
      zpages: {}

Configuring the service section and data pipelines

All the components that are configured must be enabled via pipelines within the service section. If a component is not defined in the service section, then it is not enabled. Pipelines make OpenTelemetry collector a must-have component in your architecture. It provides the flexibility of receiving and exporting data in multiple formats.

In the example shown below from SigNoz OTel Collector config file, we have enabled two pipelines.

1. traces
   In this pipeline, we can receive traces in jaeger and otlp formats. We then use three processors on the collected traces, namely signozspanmetrics/prometheus and batch. We export the processed traces to write to in ClickHouse DB.

2. metrics
   In the pipeline, we receive metrics in otlp formats. Process the collected metrics using batch processor and then export the processed metrics to ClickHouse.

service:
      extensions: [health_check, zpages]
      pipelines:
        traces:
          receivers: [jaeger, otlp]
          processors: [signozspanmetrics/prometheus, batch]
          exporters: [clickhousetraces]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [clickhousemetricswrite]

A sample OpenTelemetry Collector configuration file having receivers for OTLP and MySQL. (Source: Monitor MySQL metrics with OpenTelemetry and SigNoz)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318
  mysql:
    endpoint: localhost:3306
    username: <your-root-username>
    password: <your-root-password>
    collection_interval: 10s
    initial_delay: 10s
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
exporters:
  otlp:
    endpoint: "ingest.{region}.signoz.cloud:443" # replace {region} with your region
    tls:
      insecure: false
    headers:
      "signoz-access-token": "{signoz-token}" # Obtain from https://{your-signoz-url}/settings/ingestion-settings
  logging:
    verbosity: detailed
service:
  telemetry:
    metrics:
      address: localhost:8888
  pipelines:
    metrics:
      receivers: [otlp, mysql]
      processors: [batch]
      exporters: [otlp]

Use Cases of OpenTelemetry Collector

OpenTelemetry Collector is a powerful tool used for gathering, processing and exporting telemetry data. Processors in OpenTelemetry Collector can be used to accomplish many real life use-cases when it comes to telemetry data processing. Some of the use-cases are:

• Prevent sensitive fields from accidentally leaking into traces. It can also ensure compliance with legal, privacy, or security requirements. (Using Redaction Processor)
• Batch telemetry data to better compress data and reduce the number of outgoing connections required to transmit the data. (Using Batch Processor)
• Modify attributes of a span, log, or metric to enrich your telemetry data. (Using attributes processor)
• Tranform your metrics, logs, or traces by renaming, adding, or deleting. You can also perform aggregations before sending the telemetry data to a backend. (Using metrics transform processor, log transform processor)

FAQs

What is OpenTelemetry agent vs collector?

OpenTelemetry Agent and OpenTelemetry Collector are two ways of deploying the OpenTelemetry Collector. We refer to it as an 'agent' when it is typically deployed close to the application, focusing on local data collection. You can also configure the agent to collect metrics about the host system using the hostmetrics receiver. Meanwhile, we refer to it as a 'collector' when it is deployed as a centralized standalone service that collects data from multiple agents or servers.

What is the difference between OpenTelemetry Collector and Jaeger?

OpenTelemetry collector is part of the OpenTelemetry project and it's designed to collect, process, and export telemetry data - including metrics, logs, and traces. While jaeger, on the other hand, is a distributed tracing system used for monitoring and troubleshooting microservices-based system. Jaeger specifically focuses on traces.

Jaeger can receive trace data from the OpenTelemetry Collector or directly from instrumented applications.

What is the difference between OpenTelemetry Collector and Prometheus?

OpenTelemetry collector is part of the OpenTelemetry project and it's designed to collect, process, and export telemetry data - including metrics, logs, and traces. While Pometheus is a specialized tool focused on time-series metrics collection, storage, and analysis.

OpenTelemetry Collector collects telemetry data from various sources, processes it, and forwards it to different backends. While Prometheus primarily collects metrics using a pull model, scraping metrics from instrumented endpoints.

What is OpenTelemetry collector-contrib?

The OpenTelemetry Collector Contrib, often referred to as "collector-contrib," is a repository and distribution of the OpenTelemetry Collector that includes a collection of components contributed by the community. These components extend the basic functionality of the OpenTelemetry Collector, offering additional flexibility and capabilities.

These contributions are not part of the core distribution of the OpenTelemetry Collector but are made available for users who need them. The contrib repo has a wide range of receivers, processors, and exporters contributed by the community.

Getting started with OpenTelemetry

OpenTelemetry provides a vendor-agnostic way of collecting and managing telemetry data. The next step is to choose a backend analysis tool that can help you make sense of the collected data. SigNoz is a full-stack open-source application performance monitoring and observability platform built natively for OpenTelemetry.

SigNoz can be used to visualize metrics and traces with charts that can enable quick insights for your teams.

SigNoz cloud is the easiest way to run SigNoz. Sign up for a free account and get 30 days of unlimited access to all features.

You can also install and self-host SigNoz yourself since it is open-source. With 19,000+ GitHub stars, open-source SigNoz is loved by developers. Find the instructions to self-host SigNoz.

Related Content

OpenTelemetry Tracing - things you need to know

OpenTelemetry Logs - A Complete Introduction & Implementation

Monitor Nodejs Application with OpenTelemetry and SigNoz