Monitor Kafka Service

Table of Contents

1. Introduction
2. Prerequisites
3. Steps to Follow
   • Step 1: Kafka Setup
   • Step 2: OpenTelemetry Java Agent Installation
   • Step 3: Java Producer-Consumer App Setup
   • Step 4: OpenTelemetry Collector Setup
   • Step 5: Visualise Data in SigNoz

Introduction

Kafka is a widely-used distributed messaging system designed to handle large volumes of real-time data. It is commonly employed for building scalable, fault-tolerant data pipelines and stream processing systems. Kafka allows producers to send messages to topics, which are partitioned to support scalability, and consumers to subscribe and process these messages asynchronously.

Key components of Kafka include:

• Producers: Applications that publish messages to Kafka topics.
• Consumers: Applications that subscribe to topics and consume messages.
• Brokers: Kafka servers that manage data flow and ensure message persistence.
• Topics: Categories to which producers send messages and consumers subscribe to.
• Partitions: Divisions within a topic to ensure scalability and parallel processing.
• Consumer Groups: Groups of consumers that allow load distribution across multiple instances.
• ZooKeeper/KRaft: Manages coordination between brokers (in older Kafka versions, ZooKeeper is used; newer versions use KRaft).

In this documentation, we will guide you through setting up Kafka monitoring with OpenTelemetry and SigNoz. You'll learn how to instrument Kafka's components—producers, consumers, topics, and brokers—so that you can collect key metrics, traces and logs. This will help you gain deep insights into the performance and behavior of your Kafka cluster.

Prerequisites

Before you begin, ensure the following prerequisites are met:

1. Java Development Kit (JDK): Ensure that JDK 8 or higher is installed on your system.
2. Apache Kafka: Kafka should be installed and running for your environment.
3. Kafka Producer and Consumer Applications: You will need Kafka producer and consumer applications in Java (Support for other languages will be added soon).
4. Kafka Client Library: Make sure the Kafka client libraries are included in your project. You can find the official Kafka client libraries from Apache here.
5. OpenTelemetry Java Agent: Download the OpenTelemetry Java auto-instrumentation agent JAR. Follow the setup guide from the official OpenTelemetry documentation here or directly download the latest agent JAR from here.

Once these are set up, you will be ready to proceed with Kafka monitoring and instrumentation.

Steps to Follow

This guide follows these primary steps:

1. Kafka Setup
2. OpenTelemetry Java Agent Installation
3. Java Producer-Consumer App Setup
4. SigNoz Setup
5. OpenTelemetry Collector Setup

# Extract the downloaded Kafka folder
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0

bin/zookeeper-server-start.sh config/zookeeper.properties

# make sure you are inside the extracted kafka folder

cp config/server.properties config/s1.properties
cp config/server.properties config/s2.properties

vi config/s1.properties

broker.id=1
listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka_logs-1
zookeeper.connect=localhost:2181

vi config/s2.properties

broker.id=2
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kafka_logs-2
zookeeper.connect=localhost:2181

JMX_PORT=2020 bin/kafka-server-start.sh config/s1.properties

JMX_PORT=2021 bin/kafka-server-start.sh config/s2.properties

bin/kafka-topics.sh --create --topic topic1 --bootstrap-server localhost:9092 --replication-factor 2 --partitions 2
bin/kafka-topics.sh --create --topic topic2 --bootstrap-server localhost:9092 --replication-factor 2 --partitions 1

bin/kafka-topics.sh --describe --topic topic1 --bootstrap-server localhost:9092
bin/kafka-topics.sh --describe --topic topic2 --bootstrap-server localhost:9092

bin/kafka-console-producer.sh --topic topic1 --bootstrap-server localhost:9092

bin/kafka-console-consumer.sh --topic topic1 --from-beginning --bootstrap-server localhost:9092

wget https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

java -javaagent:/path/to/opentelemetry-javaagent.jar \
     -Dotel.service.name=producer-svc \
     -Dotel.traces.exporter=otlp \
     -Dotel.metrics.exporter=otlp \
     -Dotel.logs.exporter=otlp \
     -jar /path/to/your/producer.jar

java -javaagent:/path/to/opentelemetry-javaagent.jar \
     -Dotel.service.name=consumer-svc \
     -Dotel.traces.exporter=otlp \
     -Dotel.metrics.exporter=otlp \
     -Dotel.logs.exporter=otlp \
     -Dotel.instrumentation.kafka.producer-propagation.enabled=true \
     -Dotel.instrumentation.kafka.experimental-span-attributes=true \
     -Dotel.instrumentation.kafka.metric-reporter.enabled=true \
     -jar /path/to/your/consumer.jar

receivers:
  # Read more about kafka metrics receiver - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkametricsreceiver/README.md
  kafkametrics:
    brokers:
      - localhost:9092
      - localhost:9093
      - localhost:9094
    protocol_version: 2.0.0
    scrapers:
      - brokers
      - topics
      - consumers
  # Read more about jmx receiver - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/jmxreceiver/README.md
  jmx/1:
    # configure the path where you installed opentelemetry-jmx-metrics jar
    jar_path: path/to/opentelemetry-jmx-metrics.jar #change this to the path to you opentelemetry-jmx-metrics jar file you downloaded above
    endpoint: service:jmx:rmi:///jndi/rmi://localhost:9991/jmxrmi
    target_system: jvm,kafka,kafka-consumer,kafka-producer
    collection_interval: 10s
    log_level: info
    resource_attributes:
      broker.name: broker1
  jmx/2:
    jar_path: ${PWD}/opentelemetry-jmx-metrics.jar
    endpoint: service:jmx:rmi:///jndi/rmi://localhost:9992/jmxrmi
    target_system: jvm,kafka,kafka-consumer,kafka-producer
    collection_interval: 10s
    log_level: info
    resource_attributes:
      broker.name: broker2
exporters:
  otlp:
    endpoint: "ingest.{region}.signoz.cloud:443"
    tls:
      insecure: false
    headers:
      "signoz-access-token": "<SIGNOZ_INGESTION_KEY>"
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [kafkametrics, jmx/1, jmx/2]
      exporters: [otlp]

./otelcol-contrib --config path/to/config.yaml

Step 5: Visualize data in SigNoz

To see your Producer and Consumer app traces:

• Head over to Services tab in your SigNoz instance.
• You should be able to see your apps in the list along with some application metrics.
• Click on the service of your choice to see more detailed application metrics and related traces.

To see the Kafka Metrics:

• Head over to Messaging Queues tab in your SigNoz instance.
• You will get different Options like Consumer Lag View etc., to see various kafka related metrics.