Skip to main content

K8s Infra Metrics

Overview

To export K8s metrics, you can enable different receivers in OpenTelemetry collector which will send metrics about your Kubernetes infrastructure to SigNoz. These OpenTelemetry collectors will act as agents which send metrics about K8s to SigNoz.

Based on where you are running SigNoz ( e.g. in an independent VM or K8s cluster), you have to provide the address to send data from the above receivers.

Steps to export K8s metrics to SigNoz

  1. Clone Otel collector repo

    git clone https://github.com/SigNoz/otel-collector-k8s.git && cd otel-collector-k8s
  2. Set up the address to SigNoz in your OTel collectors
    You need to set up the address to SigNoz in your OTel collector which is collecting the K8s metrics.

    a. If you are running SigNoz in an independent VM, you need to change the placeholder IPs in the following files with the IP of machine where you are hosting signoz.

    You need to update the below section.

    exporters:
    otlp:
    endpoint: "<IP of machine hosting SigNoz>:4317"
    tls:
    insecure: true

    b. If you are running SigNoz in the same Kubernetes cluster where your applications are, you have to replace the above endpoint in agent/infra-metrics.yaml and deployment/all-in-one.yaml by

    <helm_release_name>-signoz-otel-collector.<namespace_hosting_signoz>.svc.cluster.local:4317
  3. Install OTel collectors and enable specific receivers to send metrics to SigNoz

    To access metrics from kubeletstats receivers you have to:

    kubectl create ns signoz-infra-metrics
    kubectl -n signoz-infra-metrics apply -Rf agent
    kubectl -n signoz-infra-metrics apply -Rf deployment
    kubectl -n signoz-infra-metrics get pods

    List of metrics from Kubernetes receiver.

  4. Plot Metrics in SigNoz UI

    To plot these metrics, follow the instructions given in the docs here.

  5. Import Dashboard with CPU and Memory Metrics

    You can import basic dashboard with CPU and Memory metrics of K8s cluster containers from here.

    After importing, you can include more widgets using other metrics from the list below.

  6. Generate and Import Dashboard with Node Metrics

    To generate node metrics dashboards for each nodes of K8s cluster:

    for node in $(kubectl get nodes -o name | sed -e "s/^node\///");
    do
    curl -sL https://github.com/SigNoz/benchmark/raw/main/dashboards/hostmetrics/hostmetrics-import.sh \
    | HOSTNAME="$node" DASHBOARD_TITLE="Node Metrics Dashboard for $node" bash
    done

    After importing the generated dashboards, you can include more widgets as per you need using metrics from the list below.


List of metrics

Kubernetes Metrics - kubeletstats and k8s_cluster
  • container_cpu_time
  • container_cpu_utilization
  • container_filesystem_available
  • container_filesystem_capacity
  • container_filesystem_usage
  • container_memory_available
  • container_memory_major_page_faults
  • container_memory_page_faults
  • container_memory_rss
  • container_memory_usage
  • container_memory_working_set
  • k8s_container_cpu_limit
  • k8s_container_cpu_request
  • k8s_container_ephemeral_storage_request
  • k8s_container_memory_limit
  • k8s_container_memory_request
  • k8s_container_ready
  • k8s_container_restarts
  • k8s_daemonset_current_scheduled_nodes
  • k8s_daemonset_desired_scheduled_nodes
  • k8s_daemonset_misscheduled_nodes
  • k8s_daemonset_ready_nodes
  • k8s_deployment_available
  • k8s_deployment_desired
  • k8s_namespace_phase
  • k8s_node_condition_memory_pressure
  • k8s_node_condition_ready
  • k8s_node_cpu_time
  • k8s_node_cpu_utilization
  • k8s_node_filesystem_available
  • k8s_node_filesystem_capacity
  • k8s_node_filesystem_usage
  • k8s_node_memory_available
  • k8s_node_memory_major_page_faults
  • k8s_node_memory_page_faults
  • k8s_node_memory_rss
  • k8s_node_memory_usage
  • k8s_node_memory_working_set
  • k8s_node_network_errors
  • k8s_node_network_io
  • k8s_pod_cpu_time
  • k8s_pod_cpu_utilization
  • k8s_pod_filesystem_available
  • k8s_pod_filesystem_capacity
  • k8s_pod_filesystem_usage
  • k8s_pod_memory_available
  • k8s_pod_memory_major_page_faults
  • k8s_pod_memory_page_faults
  • k8s_pod_memory_rss
  • k8s_pod_memory_usage
  • k8s_pod_memory_working_set
  • k8s_pod_phase
  • k8s_replicaset_available
  • k8s_replicaset_desired
  • k8s_statefulset_current_pods
  • k8s_statefulset_desired_pods
  • k8s_statefulset_ready_pods
  • k8s_statefulset_updated_pods
Node Metrics - hostmetrics
  • system_network_connections
  • system_disk_weighted_io_time
  • system_disk_merged
  • system_disk_operation_time
  • system_disk_pending_operations
  • system_disk_io_time
  • system_disk_operations
  • system_disk_io
  • system_filesystem_inodes_usage
  • system_filesystem_usage
  • system_cpu_time
  • system_memory_usage
  • system_network_packets
  • system_network_dropped
  • system_network_io
  • system_network_errors
  • system_cpu_load_average_5m
  • system_cpu_load_average_15m
  • system_cpu_load_average_1m