Overview
Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed Kafka service. SigNoz helps you monitor broker health, consumer lag, and throughput.
Prerequisites
- AWS account with appropriate permissions
- SigNoz Cloud account or Self-Hosted SigNoz
One-Click Integration
One-Click Integration is available for SigNoz Cloud only and includes pre-built dashboards. This method uses AWS CloudFormation and CloudWatch, which may incur additional AWS charges.
Step 1: Connect Your AWS Account
Follow the One-Click AWS Integrations Guide to:
- Deploy the CloudFormation stack
- Connect your AWS account to SigNoz
Step 2: Enable MSK Monitoring
Once connected, SigNoz will auto-discover your MSK clusters and begin collecting CloudWatch metrics.
What's Collected
| Metric | Description |
|---|---|
BytesInPerSec | Incoming bytes per second |
BytesOutPerSec | Outgoing bytes per second |
MessagesInPerSec | Incoming messages per second |
PartitionCount | Number of partitions |
UnderReplicatedPartitions | Under-replicated partitions |
OfflinePartitionsCount | Offline partitions |
ActiveControllerCount | Active controller count |
Pre-built Dashboards
Navigate to Dashboards and search for "MSK" to find automatically imported dashboards.
Manual Setup
Manual setup works for both SigNoz Cloud and Self-Hosted. You'll need to set up your own dashboards.
For comprehensive MSK monitoring with consumer lag tracking and traces, see the detailed guide:
This guide covers:
- Enabling JMX/Node exporter endpoints on MSK
- Configuring OpenTelemetry Collector to scrape broker metrics
- Instrumenting Java producer/consumer applications with OTel Java Agent
- Visualizing data in SigNoz Messaging Queues tab
MSK Monitoring Architecture
MSK Cluster (JMX:11001, Node:11002)
│
▼
OTel Collector (Prometheus receiver)
│
▼
SigNoz
