Skip to main content

K8s Infra Metrics and Logs Collection

Overview

To export K8s metrics, you can enable different receivers in OpenTelemetry collector which will send metrics about your Kubernetes (K8s) infrastructure to SigNoz. These OpenTelemetry collectors will act as agents which send metrics about K8s to SigNoz.

OtelCollector agent can also be used to tail and parse logs generated by container using filelog receiver and send it to desired receiver.

Based on where you are running SigNoz ( e.g. in an independent VM or K8s cluster), you have to provide the address to send data from the above receivers.

Steps to export K8s metrics to SigNoz

By default, SigNoz Helm chart installs k8s-infra dependency chart which handles the task of collecting metrics and logs from the K8s cluster.

K8s-Infra helm chart mainly handles the following:

  • Tails and parses logs generated by containers in K8s cluster and sends to SigNoz
  • Collects kubelet metrics and host metrics from each nodes of the K8s cluster
  • Collects cluster-level metrics from the Kubernetes API server
  • Acts as a gateway to send any incoming OTLP telemetry data to SigNoz OtelCollector
info

Skip Step 1 if you have single K8s cluster for SigNoz and your applications as k8s-infra chart is included in default SigNoz chart installation.

  1. Install K8s-Infra chart

    helm install my-release signoz/k8s-infra  \
    --set otelCollectorEndpoint=<IP-or-Endpoint-of-SigNoz-OtelCollector>:4317
    info

    If the OtelCollector endpoint is secured, you would have to enable otelInsecure configuration and often make other changes such as including either config or path to the TLS certificate and private key.

    In case of SigNoz Cloud, you would have to set signozApiKey configuration.

  2. Plot Metrics in SigNoz UI

    To plot metrics generated from k8s-infra chart, follow the instructions given in the docs here.

    Check out the List of metrics from Kubernetes receiver.

  3. Import Dashboard with CPU and Memory Metrics

    You can import basic dashboard with CPU and Memory metrics of K8s cluster containers from here.

    After importing, you can include more widgets using other metrics from the list below.

  4. Generate and Import Dashboard with Node Metrics

    To generate node metrics dashboards for each nodes of K8s cluster:

    for node in $(kubectl get nodes -o name | sed -e "s/^node\///");
    do
    curl -sL https://github.com/SigNoz/benchmark/raw/main/dashboards/hostmetrics/hostmetrics-import.sh \
    | HOSTNAME="$node" DASHBOARD_TITLE="Node Metrics Dashboard for $node" bash
    done

    After importing the generated dashboards, you can include more widgets as per you need using metrics from the list below.


List of metrics

Kubernetes Metrics - kubeletstats and k8s_cluster
  • container_cpu_time
  • container_cpu_utilization
  • container_filesystem_available
  • container_filesystem_capacity
  • container_filesystem_usage
  • container_memory_available
  • container_memory_major_page_faults
  • container_memory_page_faults
  • container_memory_rss
  • container_memory_usage
  • container_memory_working_set
  • k8s_container_cpu_limit
  • k8s_container_cpu_request
  • k8s_container_ephemeral_storage_request
  • k8s_container_memory_limit
  • k8s_container_memory_request
  • k8s_container_ready
  • k8s_container_restarts
  • k8s_daemonset_current_scheduled_nodes
  • k8s_daemonset_desired_scheduled_nodes
  • k8s_daemonset_misscheduled_nodes
  • k8s_daemonset_ready_nodes
  • k8s_deployment_available
  • k8s_deployment_desired
  • k8s_namespace_phase
  • k8s_node_condition_memory_pressure
  • k8s_node_condition_ready
  • k8s_node_cpu_time
  • k8s_node_cpu_utilization
  • k8s_node_filesystem_available
  • k8s_node_filesystem_capacity
  • k8s_node_filesystem_usage
  • k8s_node_memory_available
  • k8s_node_memory_major_page_faults
  • k8s_node_memory_page_faults
  • k8s_node_memory_rss
  • k8s_node_memory_usage
  • k8s_node_memory_working_set
  • k8s_node_network_errors
  • k8s_node_network_io
  • k8s_pod_cpu_time
  • k8s_pod_cpu_utilization
  • k8s_pod_filesystem_available
  • k8s_pod_filesystem_capacity
  • k8s_pod_filesystem_usage
  • k8s_pod_memory_available
  • k8s_pod_memory_major_page_faults
  • k8s_pod_memory_page_faults
  • k8s_pod_memory_rss
  • k8s_pod_memory_usage
  • k8s_pod_memory_working_set
  • k8s_pod_phase
  • k8s_replicaset_available
  • k8s_replicaset_desired
  • k8s_statefulset_current_pods
  • k8s_statefulset_desired_pods
  • k8s_statefulset_ready_pods
  • k8s_statefulset_updated_pods
Node Metrics - hostmetrics
  • system_network_connections
  • system_disk_weighted_io_time
  • system_disk_merged
  • system_disk_operation_time
  • system_disk_pending_operations
  • system_disk_io_time
  • system_disk_operations
  • system_disk_io
  • system_filesystem_inodes_usage
  • system_filesystem_usage
  • system_cpu_time
  • system_memory_usage
  • system_network_packets
  • system_network_dropped
  • system_network_io
  • system_network_errors
  • system_cpu_load_average_5m
  • system_cpu_load_average_15m
  • system_cpu_load_average_1m