This integration enables monitoring of your Amazon Managed Streaming for Apache Kafka (MSK) clusters by collecting comprehensive Kafka metrics. With SigNoz's one-click AWS integration, you can track broker health, throughput, and partition status.
Amazon MSK is a fully managed Apache Kafka service. Monitoring MSK is essential for understanding cluster health, message throughput, and ensuring reliable data streaming.
Getting Started
To use the MSK integration:
- From the sidebar, navigate to Integrations.
- Click Configure on the Amazon Web Services tile.
- Navigate to MSK to view dashboards and configure the integration.
For detailed setup instructions, see One-Click AWS Integrations.
What You'll Monitor & Data Collected
This integration collects comprehensive MSK metrics, providing visibility into:
Core MSK Metrics
- Throughput: Monitor bytes in/out per second.
- Controller Status: Track active controller count.
- Partition Health: Monitor offline and under-replicated partitions.
- Leader Count: Track broker leader counts.
- Consumer Lag: Monitor offset lag.
- Resource Usage: Track CPU, memory, and disk usage.
Collected Data Details
📊 Metrics List
| Metric Name | Unit | Type |
|---|---|---|
| aws_Kafka_ActiveControllerCount_count | Count | Gauge |
| aws_Kafka_ActiveControllerCount_max | Count | Gauge |
| aws_Kafka_ActiveControllerCount_min | Count | Gauge |
| aws_Kafka_ActiveControllerCount_sum | Count | Gauge |
| aws_Kafka_BytesInPerSec_count | Bytes/Second | Gauge |
| aws_Kafka_BytesInPerSec_max | Bytes/Second | Gauge |
| aws_Kafka_BytesInPerSec_min | Bytes/Second | Gauge |
| aws_Kafka_BytesInPerSec_sum | Bytes/Second | Gauge |
| aws_Kafka_BytesOutPerSec_count | Bytes/Second | Gauge |
| aws_Kafka_BytesOutPerSec_max | Bytes/Second | Gauge |
| aws_Kafka_BytesOutPerSec_min | Bytes/Second | Gauge |
| aws_Kafka_BytesOutPerSec_sum | Bytes/Second | Gauge |
| aws_Kafka_CpuIdle_count | Percent | Gauge |
| aws_Kafka_CpuIdle_max | Percent | Gauge |
| aws_Kafka_CpuIdle_min | Percent | Gauge |
| aws_Kafka_CpuIdle_sum | Percent | Gauge |
| aws_Kafka_LeaderCount_count | Count | Gauge |
| aws_Kafka_LeaderCount_max | Count | Gauge |
| aws_Kafka_LeaderCount_min | Count | Gauge |
| aws_Kafka_LeaderCount_sum | Count | Gauge |
| aws_Kafka_MaxOffsetLag_count | Count | Gauge |
| aws_Kafka_MaxOffsetLag_max | Count | Gauge |
| aws_Kafka_MaxOffsetLag_min | Count | Gauge |
| aws_Kafka_MaxOffsetLag_sum | Count | Gauge |
| aws_Kafka_MemoryUsed_count | Bytes | Gauge |
| aws_Kafka_MemoryUsed_max | Bytes | Gauge |
| aws_Kafka_MemoryUsed_min | Bytes | Gauge |
| aws_Kafka_MemoryUsed_sum | Bytes | Gauge |
| aws_Kafka_OfflinePartitionsCount_count | Count | Gauge |
| aws_Kafka_OfflinePartitionsCount_max | Count | Gauge |
| aws_Kafka_OfflinePartitionsCount_min | Count | Gauge |
| aws_Kafka_OfflinePartitionsCount_sum | Count | Gauge |
| aws_Kafka_UnderReplicatedPartitions_count | Count | Gauge |
| aws_Kafka_UnderReplicatedPartitions_max | Count | Gauge |
| aws_Kafka_UnderReplicatedPartitions_min | Count | Gauge |
| aws_Kafka_UnderReplicatedPartitions_sum | Count | Gauge |
| aws_Kafka_BurstBalance_count | Percent | Gauge |
| aws_Kafka_BurstBalance_max | Percent | Gauge |
| aws_Kafka_BurstBalance_min | Percent | Gauge |
| aws_Kafka_BurstBalance_sum | Percent | Gauge |
Dashboards
MSK Overview Dashboard
The integration provides a comprehensive overview dashboard with key MSK metrics:
Key visualizations include:
- Bytes In/Out: Monitor message throughput.
- Partition Health: Track under-replicated and offline partitions.
- Leader Count: Monitor broker role distribution.
- Resource Usage: Track CPU and memory utilization.
