Setup Kubernetes Metrics with OpenTelemetry

Kubernetes Monitoring Configuration Guide

This guide explains how to set up monitoring for your Kubernetes cluster using two essential OpenTelemetry receivers:

• kubeletstats: Collects metrics from individual Kubernetes nodes
• k8scluster: Gathers cluster-wide metrics

Before you start: install the Kubernetes Collection Agent

Make sure you have an OpenTelemetry Collector running in your cluster. Pick the install guide that matches your setup:

• Standard Kubernetes → k8s-infra Helm chart: Helm chart that already wires kubeletstats, k8scluster, and k8sattributes.
• EKS Fargate / GKE Autopilot / AKS Virtual Nodes → Serverless Kubernetes Collection Agent: for managed clusters that block kubelet scraping on worker nodes.
• OpenTelemetry Operator → OTel Operator Collection Agent: if you're already managing collectors via the OpenTelemetry Operator.

Every install path ships a config that already includes the receivers and processors covered below. The sections that follow explain the receiver / processor options so you can adjust the shipped config for your needs.

Setup Kubelet Stats Receiver

Overview

The Kubelet Stats Receiver collects four types of metrics:

• Node metrics (CPU, memory, etc.)
• Pod metrics
• Container metrics
• Volume metrics

These metrics are collected directly from the Kubelet API server running on each node.

Quick Start with k8s-infra Helm Chart

The simplest way to get started is using the k8s-infra Helm chart, which automatically:

1. Deploys agents to all cluster nodes
2. Configures secure API access with appropriate RBAC permissions
3. Enables essential metrics by default

Basic configuration example:

kubeletMetrics:
  collectionInterval: 60s  # Collect metrics every minute
  metrics:
    k8s.pod.cpu_request_utilization:
      enabled: true  # Enable specific optional metrics

Advanced Metrics Configuration

The receiver supports many optional metrics that provide deeper insights into cluster efficiency. Key metrics include:

• k8s.node.cpu.usage: Tracks CPU usage on the node
• k8s.node.uptime: Tracks node uptime
• k8s.pod.cpu.usage: Tracks CPU usage on the pod
• k8s.pod.cpu_limit_utilization: Tracks CPU usage relative to configured limits
• k8s.pod.cpu_request_utilization: Measures actual CPU usage against requested resources
• k8s.pod.memory_limit_utilization: Tracks memory usage relative to configured limits
• k8s.pod.memory_request_utilization: Monitors memory usage versus requests
• k8s.pod.uptime: Tracks pod stability
• container.cpu.usage: Tracks CPU usage on the container
• k8s.container.cpu_limit_utilization: Tracks CPU usage relative to configured limits
• k8s.container.cpu_request_utilization: Measures actual CPU usage against requested resources
• k8s.container.memory_limit_utilization: Tracks memory usage relative to configured limits
• k8s.container.memory_request_utilization: Monitors memory usage versus requests
• container.uptime: Tracks container uptime
• k8s.container.cpu.node.utilization: Container CPU usage as a share of the node's total capacity
• k8s.container.memory.node.utilization: Container memory usage as a share of the node's total capacity

These are enabled by default in the k8s-infra Helm chart. For a complete list of available optional metrics, see the OpenTelemetry documentation.

Manual Configuration Guide

If you're not using the k8s-infra Helm chart, follow these steps for manual setup:

1. Configure Pod Access to Kubelet API

Add the node name to your pod specification:

env:
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

2. Configure the Kubelet Stats Receiver

Add this configuration to your collector:

receivers:
  kubeletstats:
    auth_type: serviceAccount
    collection_interval: 30s
    endpoint: ${env:K8S_NODE_NAME}:10250
    extra_metadata_labels:
      - container.id
      - k8s.volume.type
    insecure_skip_verify: true
    # Required for the *.node.utilization metrics: with `node` set (below), the receiver
    # reads the node's total capacity from the K8s API. Needs `get` on `nodes` for the service account.
    k8s_api_config:
      auth_type: serviceAccount
    metric_groups:
      - container
      - pod
      - node
      - volume
    metrics:
      container.cpu.usage:
        enabled: true
      container.uptime:
        enabled: true
      k8s.container.cpu_limit_utilization:
        enabled: true
      k8s.container.cpu_request_utilization:
        enabled: true
      k8s.container.memory_limit_utilization:
        enabled: true
      k8s.container.memory_request_utilization:
        enabled: true
      k8s.container.cpu.node.utilization:
        enabled: true
      k8s.container.memory.node.utilization:
        enabled: true
      k8s.node.cpu.usage:
        enabled: true
      k8s.node.uptime:
        enabled: true
      k8s.pod.cpu.usage:
        enabled: true
      k8s.pod.cpu_limit_utilization:
        enabled: true
      k8s.pod.cpu_request_utilization:
        enabled: true
      k8s.pod.memory_limit_utilization:
        enabled: true
      k8s.pod.memory_request_utilization:
        enabled: true
      k8s.pod.uptime:
        enabled: true
    node: ${env:K8S_NODE_NAME}

The k8s.container.{cpu,memory}.node.utilization metrics need the receiver to read the node's total capacity from the Kubernetes API. This requires node set to the node name and k8s_api_config configured (both shown above), and the collector's service account must have get permission on nodes. See the kubeletstats receiver documentation.

3. Setup k8sattributesprocessor to enable Kubernetes metadata

Configure the k8sattributes processor to enrich your metrics with Kubernetes context:

processors:
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.job.name
        - k8s.node.name
        - k8s.node.uid
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.pod.start_time
        - k8s.container.name
    filter:
      node_from_env_var: K8S_NODE_NAME
    passthrough: false
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: connection

4. Pipeline Configuration

Ensure your pipeline includes both the kubeletstats receiver and k8sattributes processor:

service:
  pipelines:
    metrics:
      receivers: [kubeletstats]
      processors: [k8sattributes]
      exporters: [otlp] # Add your exporters here

Read more about the kubeletstats receiver and k8sattributes processor.

Setup K8s Cluster Receiver

Overview

The k8scluster receiver provides cluster-wide metrics by collecting data directly from the Kubernetes API server. Unlike the kubeletstats receiver which runs on each node, you only need a single instance of this receiver per cluster.

Quick Start with k8s-infra Helm Chart

The k8s-infra Helm chart provides a production-ready configuration by:

1. Deploying a single, properly configured k8scluster receiver
2. Setting up secure API access with appropriate RBAC permissions
3. Enabling commonly needed metrics by default

Basic configuration example:

clusterMetrics:
  collectionInterval: 60s  # Collect metrics every minute
  metrics:
    k8s.node.condition:
      enabled: true  # Enable condition monitoring

Core Monitoring Features

1. Node Conditions

Monitor the health and status of your nodes with these key conditions:

• Ready: Node's ability to accept pods (enabled by default)
• MemoryPressure: Memory resource constraints
• DiskPressure: Storage resource constraints
• PIDPressure: Process ID availability
• NetworkUnavailable: Network configuration status

2. Resource Allocation Tracking

Monitor resource allocation across your cluster:

• cpu: CPU resource allocation
• memory: Memory resource allocation
• ephemeral-storage: Temporary storage allocation
• storage: Persistent storage allocation

These allocations are exposed via the k8s.node.allocatable_cpu, k8s.node.allocatable_memory, etc. metrics. They are disabled by default in the upstream k8scluster receiver. Opt in via the allocatable_types_to_report config option (shown in the manual configuration example below).

3. Optional Metrics

Key optional metrics for enhanced monitoring:

• k8s.node.condition: Detailed node health status
• k8s.pod.status_reason: Pod-level status reasons (Evicted, NodeLost, Shutdown, ...)
• k8s.container.status.reason: Container-level status reasons (CrashLoopBackOff, OOMKilled, ImagePullBackOff, ...)
• k8s.container.status.state: Container lifecycle state (running, waiting, terminated)

k8s.pod.status_reason, k8s.container.status.reason, and k8s.container.status.state are disabled by default in the upstream receiver but enabled by default in the k8s-infra Helm chart. k8s.pod.status_reason and k8s.container.status.reason drive the kubectl-style Pod Status column, while k8s.container.status.reason and k8s.container.status.state together drive the Status column on the Containers view. Enable them manually only if you run the k8s_cluster receiver without the chart (or on a chart version older than the one that ships them on by default).

For a comprehensive list of available metrics, conditions, and resources, refer to the k8sclusterreceiver documentation.

Manual Configuration Guide

If you're not using the k8s-infra Helm chart, follow this example:

1. Configure the K8s Cluster Receiver

receivers:
  k8scluster:
    collection_interval: 30s
    node_conditions_to_report: [Ready, DiskPressure, MemoryPressure, PIDPressure, NetworkUnavailable]
    allocatable_types_to_report: [cpu, memory, ephemeral-storage]
    distribution: kubernetes
    metrics:
      k8s.node.condition:
        enabled: true
      k8s.pod.status_reason:
        enabled: true
      k8s.container.status.reason:
        enabled: true
      k8s.container.status.state:
        enabled: true

2. Setup k8sattributesprocessor to enable Kubernetes metadata

Configure the k8sattributes processor to add Kubernetes context:

processors:
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.statefulset.name
        - k8s.daemonset.name
        - k8s.cronjob.name
        - k8s.job.name
        - k8s.node.name
        - k8s.node.uid
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.pod.start_time
        - k8s.container.name
    passthrough: false
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: connection

3. Pipeline Configuration

Set up the metrics pipeline:

service:
  pipelines:
    metrics:
      receivers: [k8scluster]
      processors: [k8sattributes]
      exporters: [otlp]  # Add your exporters here

Best Practices

1. Start with the k8s-infra Helm chart if possible - it provides a tested, sensible default configuration
2. Enable optional metrics for deeper visibility into your cluster
3. Use the k8sattributes processor to add context to your metrics