K8s Infra - Configure

The install guide provides a working config: the agent collects host and kubelet metrics, container logs, and data from instrumented applications on every node, and the deployment collector collects cluster metrics and Kubernetes events. This page explains what each component in that config does and how to customize it, through the values of the k8s-infra Helm chart or through Foundry's casting.yaml.

With the Helm CLI, you can use the presets in the k8s-infra Helm chart to configure the collectors, and the chart exposes the full OpenTelemetry Collector configuration via Helm values for everything else.

The presets section in the values.yaml file allows you to enable or disable specific configurations. To check the available presets, refer to the k8s-infra Helm chart documentation.

Prerequisites

K8s Infra Collection Agent installed with the Helm CLI or with Foundry (see Installation guide)
SigNoz Cloud account or self-hosted SigNoz instance
Access to the cluster with kubectl, and the files you installed from: override-values.yaml and Helm 3 for the Helm CLI, or casting.yaml and foundryctl for Foundry

Apply Changes

For each change:

Add the section's values to override-values.yaml.
Run this from the directory that contains override-values.yaml to apply the change:
```
helm upgrade my-release signoz/k8s-infra -f override-values.yaml
```
Helm updates the release, and the collector pods restart with the new config.
Run this on the machine where you use kubectl to check that the pods restart:
```
kubectl get pods
```
Then view the collector logs to confirm there are no configuration errors:
```
kubectl logs daemonset/my-release-k8s-infra-otel-agent --since=5m
```
For the deployment collector, use deployment/my-release-k8s-infra-otel-deployment.
In SigNoz, confirm data in Infrastructure Monitoring for node, pod, and container metrics, Logs for container logs and Kubernetes events, and Traces for application traces.

For each change:

Add the section's snippet to casting.yaml, in the agent's document or in the deployment collector's document (collector.kind: deployment). Foundry already generates the receivers and processors of the default presets, so add only what you change.
Run this from the directory that contains casting.yaml to apply the change:
```
foundryctl cast -f casting.yaml
```
Foundry updates both collectors, and the collector pods restart with the new config.
Run this on the machine where you use kubectl to check that the pods restart:
```
kubectl get pods -n signoz
```
Then view the collector logs to confirm there are no configuration errors:
```
kubectl logs -n signoz daemonset/signoz-collector-agent --since=5m
```
For the deployment collector, use deployment/signoz-collector-deployment.
In SigNoz, confirm data in Infrastructure Monitoring for node, pod, and container metrics, Logs for container logs and Kubernetes events, and Traces for application traces.

Send Data from Applications to the Collector

Data flow from your application to SigNoz — *OpenTelemetry instrumented application sends data to OTelAgent Daemon deployed in your k8s infra. The OTelAgent daemon sends the collected data to SigNoz.*

Node, pod, and container metrics, container logs, and Kubernetes events flow as soon as the agent pods are running.

Where and how to send data from your application

To send traces, metrics, and logs from an application, instrument it with OpenTelemetry (language guides), then add this to the application's container in its manifest:

env:
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: K8S_POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: K8S_POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://$(HOST_IP):4318
  - name: OTEL_SERVICE_NAME
    value: <service-name>
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.pod.ip=$(K8S_POD_IP),k8s.pod.uid=$(K8S_POD_UID)

Verify these values:

<service-name>: The name your application shows under in SigNoz.

The application sends to the agent on its own node, which adds the Kubernetes resource attributes, matched by k8s.pod.ip and k8s.pod.uid, and forwards everything to SigNoz.

The agent binds no host ports, so set OTEL_EXPORTER_OTLP_ENDPOINT to the agent's Service instead:

http://my-release-k8s-infra-otel-agent.<namespace>.svc.cluster.local:4318 for a Helm CLI install (replace my-release and <namespace> with your release and namespace).
http://signoz-collector-agent.signoz.svc.cluster.local:4318 for a Foundry install (signoz is metadata.name, which is also the namespace unless the foundry.signoz.io/kubernetes-namespace annotation sets another).

Redeploy the application and send it some traffic. Within a minute it shows under Services in SigNoz, and its traces carry k8s.namespace.name and k8s.pod.name.

Export to SigNoz

The OTLP/HTTP exporter sends the telemetry that both collectors gather to SigNoz.

presets:
  otlphttpExporter:
    enabled: true # Must be enabled for data to reach SigNoz

What it does: Enables sending telemetry data to SigNoz backend over OTLP/HTTP. This is required for SigNoz to receive any data.

Collect Host Metrics

Use the hostmetrics receiver to scrape CPU, memory, disk, filesystem, load, and network metrics from each node.

presets:
  hostMetrics:
    enabled: true
    collectionInterval: 30s # Adjust based on your monitoring needs
    scrapers:
      cpu: {} # Collects CPU usage, time, and utilization metrics
      load: {} # Collects system load averages
      memory: {} # Collects memory usage metrics
      disk: # Collects disk I/O metrics
        exclude:
          devices: # Regex patterns to exclude specific devices
            - ^ram\d+$
            - ^loop\d+$
            # Add custom device exclusions here
      filesystem: # Collects filesystem metrics
        exclude_fs_types:
          fs_types:
            - tmpfs
            - squashfs
            # Add filesystem types to exclude
      network: # Collects network metrics
        exclude:
          interfaces:
            - ^veth.*$ # Excludes virtual ethernet devices
            - ^docker.*$ # Excludes docker interfaces
            # Add network interfaces to exclude

Key points:

Adjust collectionInterval based on your monitoring granularity needs
Customize device exclusions to reduce noise from irrelevant storage devices
Configure network interface exclusions to focus on relevant network metrics

Foundry collects host metrics by default, every 30s. In casting.yaml, add this to the agent's document to change the interval and the excluded disk devices:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics:
                collection_interval: 60s
                scrapers:
                  disk:
                    exclude:
                      devices: [^ram\d+$, ^loop\d+$]

Lists replace, so devices holds only the entries you list. Foundry's default list is ^ram\d+$, ^zram\d+$, ^loop\d+$, ^fd\d+$, ^hd[a-z]\d+$, ^sd[a-z]\d+$, ^vd[a-z]\d+$, ^xvd[a-z]\d+$ and ^nvme\d+n\d+p\d+$; restate the ones you keep. The filesystem and network exclusion lists work the same way.

Collect Container Logs

Use the filelog receiver to tail the container log files on each node. For full setup details, see Collecting Kubernetes Pod Logs.

presets:
  logsCollection:
    enabled: true
    startAt: beginning # Options: beginning or end
    includeFilePath: true # Adds file path as metadata
    includeFileName: false # Adds file name as metadata
 
    # Exclusion configuration
    blacklist:
      enabled: true
      signozLogs: true # Excludes SigNoz's own logs
      namespaces:
        - kube-system # Add namespaces to exclude
      pods:
        - hotrod # Add pod names to exclude
        - locust
      containers: [] # Add container names to exclude
 
    # Inclusion configuration (overrides blacklist if enabled)
    whitelist:
      enabled: false
      signozLogs: true
      namespaces: [] # Only collect logs from these namespaces
      pods: [] # Only collect logs from these pods
      containers: [] # Only collect logs from these containers

Best practices:

Use blacklist for excluding noisy system pods
Enable whitelist when you need to monitor specific applications only
Consider disk usage implications when enabling file path/name inclusion

Foundry collects container logs by default, from the end of each file, without the collectors' own logs and kube-system. In casting.yaml, add this to the agent's document to read from the beginning and exclude more pods:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/k8s:
                start_at: beginning
                exclude:
                  - /var/log/pods/signoz_signoz-collector-*/*/*.log
                  - /var/log/pods/kube-system_*/*/*.log
                  - /var/log/pods/*_hotrod*_*/*/*.log
                  - /var/log/pods/*_locust*_*/*/*.log

Lists replace, so restate Foundry's two default exclusions (the first two entries) before your own. The first one skips the collectors' own logs and follows <namespace>_<metadata.name>-collector-*. Exclude a namespace with /var/log/pods/<namespace>_*/*/*.log, a pod with /var/log/pods/*_<pod>*_*/*/*.log, or a container with /var/log/pods/*_*_*/<container>/*.log.

To collect logs from some namespaces only, replace include:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/k8s:
                include:
                  - /var/log/pods/production_*/*/*.log

Disable Logs Collection

In case you do not want to collect logs from your Kubernetes cluster, you can disable using presets in k8s-infra chart.

presets:
  logsCollection:
    enabled: false

In casting.yaml, add this to the agent's document. It removes the filelog/k8s receiver; logs that applications send over OTLP keep flowing:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/k8s: null
            service:
              pipelines:
                logs:
                  receivers: [otlp/http, otlp/grpc]

null removes the receiver. Lists replace, so the logs pipeline is restated with Foundry's two OTLP receivers.

Collect Kubelet Metrics

Use the kubeletstats receiver to collect node, pod, container, and volume metrics from the kubelet on each node.

presets:
  kubeletMetrics:
    enabled: true
    collectionInterval: 30s
    authType: serviceAccount # Authentication method
    endpoint: ${env:K8S_HOST_IP}:10250
 
    # Metrics configuration
    metrics:
      k8s.pod.cpu_limit_utilization:
        enabled: true
      k8s.pod.memory_limit_utilization:
        enabled: true
      # Container-level metrics: power the Containers view
      k8s.container.cpu_request_utilization:
        enabled: true
      k8s.container.cpu_limit_utilization:
        enabled: true
      k8s.container.memory_request_utilization:
        enabled: true
      k8s.container.memory_limit_utilization:
        enabled: true
      container.uptime:
        enabled: true
      k8s.container.cpu.node.utilization:
        enabled: true
      k8s.container.memory.node.utilization:
        enabled: true
      # Enable other metrics as needed

Important settings:

Set appropriate collectionInterval based on cluster size
Enable specific metrics based on monitoring requirements

Foundry collects kubelet metrics by default, every 30s, with the metrics in the table below enabled. In casting.yaml, add this to the agent's document to change the interval, verify the kubelet's certificate, and enable one more metric:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              kubeletstats:
                collection_interval: 60s
                insecure_skip_verify: false
                metrics:
                  k8s.pod.memory.node.utilization:
                    enabled: true

metrics is a map, so the metrics you add join Foundry's defaults. With insecure_skip_verify: false, the receiver checks the kubelet's certificate against the cluster CA.

These container-level metrics are disabled by default in the upstream kubeletstats receiver but enabled by default in this chart and by Foundry. Keep them enabled to retain full container visibility on the Containers view:

Metric	Powers
`k8s.container.cpu_request_utilization`	CPU request utilization column
`k8s.container.cpu_limit_utilization`	CPU limit utilization column
`k8s.container.memory_request_utilization`	Memory request utilization column
`k8s.container.memory_limit_utilization`	Memory limit utilization column
`container.uptime`	Uptime chart
`k8s.container.cpu.node.utilization`	Node-utilization chart (CPU)
`k8s.container.memory.node.utilization`	Node-utilization chart (memory)

Collect Cluster Metrics

Use the k8s_cluster receiver on the deployment collector to collect cluster-level metrics from the Kubernetes API server.

presets:
  clusterMetrics:
    enabled: true
    collectionInterval: 30s
 
    # Node conditions to monitor
    nodeConditionsToReport:
      - Ready
      - MemoryPressure
      - DiskPressure
      # Add conditions based on monitoring needs
 
    # Resource types to monitor
    allocatableTypesToReport:
      - cpu
      - memory
      # - storage    # Uncomment if needed
 
    # Fine-grained metric enablement
    metrics:
      k8s.pod.status_reason:
        enabled: true
      k8s.container.status.reason:
        enabled: true
      k8s.container.status.state:
        enabled: true

When to use:

Enable for cluster health monitoring
Useful for capacity planning and resource optimization
Essential for multi-node cluster monitoring

Foundry collects cluster metrics by default from the deployment collector, every 30s, with the metrics in the table below enabled. In casting.yaml, add this to the deployment collector's document to change the interval and report storage too:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              k8s_cluster:
                collection_interval: 60s
                allocatable_types_to_report: [cpu, memory, storage]

Lists replace, so restate every entry you keep. Foundry's defaults are allocatable_types_to_report: [cpu, memory] and node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable].

These metrics are disabled by default in the upstream k8s_cluster receiver but enabled by default in this chart and by Foundry. Keep them enabled to retain pod- and container-status visibility. Without them, status falls back to the lifecycle phase or state only:

Metric	Powers
`k8s.pod.status_reason`	Pods view kubectl-style Pod Status column (and the per-entity Pod Counts by Status breakdown)
`k8s.container.status.reason`	Pods view Pod Status column; Containers view Status column
`k8s.container.status.state`	Containers view Status column

k8s.container.status.reason and k8s.container.status.state are experimental (development stability) upstream, so their allowed values may change on a collector upgrade.

Collect Kubernetes Events

Use the k8s_events receiver on the deployment collector to collect Kubernetes events as logs.

presets:
  k8sEvents:
    enabled: true
    authType: serviceAccount
    namespaces: [] # Empty for all namespaces, or specify list

Usage scenarios:

Enable for debugging and audit trails
Monitor specific namespaces by listing them
Useful for compliance and security monitoring

Foundry collects Kubernetes events by default from the deployment collector, for all namespaces. In casting.yaml, add this to the deployment collector's document to watch specific namespaces:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              k8s_events:
                namespaces: [default, production]

Resource Detection and Kubernetes Attributes

The resourcedetection processor adds host and cloud resource attributes, and the k8sattributes processor adds Kubernetes resource attributes, so you can filter and group by namespace, workload, or pod in SigNoz.

presets:
  kubernetesAttributes:
    enabled: true
    passthrough: false # Set true to disable k8s API calls
 
    # Control which metadata to collect
    extractMetadatas:
      - k8s.namespace.name
      - k8s.deployment.name
      - k8s.pod.name
      - k8s.node.name
      # Add or remove metadata fields
 
    # Pod association configuration
    podAssociation:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: connection
    # This label/annotation extraction rule takes the value of the key from labels/annotations and maps it to the tag_name attribute which will be added to the associated resources
    extractLabels:
      # Example from pod
      - tag_name: service.name
        key: app.kubernetes.io/name
        from: pod
      # Example from node
      - tag_name: os.type
        key: beta.kubernetes.io/os
        from: node
 
    # Annotation extraction configuration
    extractAnnotations:
      # Example from node
      - tag_name: node.ttl
        key: node.alpha.kubernetes.io/ttl
        from: node

Configuration tips:

List only the resource attributes you need under extractMetadatas to optimize performance
Use passthrough: true in very large clusters to reduce API load
Configure pod association based on your networking setup

Foundry enriches telemetry by default with the k8sattributes processor, with the chart's default extract.metadata attributes and pod association rules. In casting.yaml, add this to the agent's document to extract a pod label and a node annotation:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              k8sattributes:
                extract:
                  labels:
                    - tag_name: service.name
                      key: app.kubernetes.io/name
                      from: pod
                  annotations:
                    - tag_name: node.ttl
                      key: node.alpha.kubernetes.io/ttl
                      from: node

Lists replace, so restating extract.metadata or pod_association drops Foundry's defaults: include every entry you keep, as the warning below describes. Set passthrough: true under k8sattributes to skip Kubernetes API lookups.

The from: field selects which Kubernetes object the processor reads the label or annotation from. The examples above use pod and node, but other workload kinds are supported as well, and each reads from its own metadata.

Refer to the OpenTelemetry k8sattributes processor documentation for the full list of supported sources and the exact behavior of each.

If telemetry suddenly stops getting Kubernetes attributes (k8s.pod.name, k8s.namespace.name, etc.) after a values.yaml change, an override on podAssociation is the most likely cause.

Helm replaces arrays entirely rather than merging them. When you override podAssociation, the chart's defaults are dropped, including the connection rule that matches telemetry by the source IP of the OTLP connection, and the k8s.pod.uid rule used by file-based log collection. Without those fallbacks the processor cannot associate telemetry with a pod, and extractLabels / extractAnnotations stop producing attributes.

If you customize podAssociation, include the rules you need from the chart defaults alongside your additions.

Scrape Prometheus Metrics from Pods

Prometheus preset allows to scrapes metrics from pods that have the appropriate annotations. This enables collecting metrics from applications that expose Prometheus endpoints without manual configuration.

presets:
  prometheus:
    enabled: true
    annotationsPrefix: signoz.io # Prefix for pod annotations
    scrapeInterval: 60s # How often to scrape metrics
    namespaceScoped: false # Limit to same namespace only
    namespaces: [] # Specific namespaces to scrape
    includePodLabel: false # Include pod labels (may increase cardinality)
    includeContainerName: false # Include container name in metrics
    scrapeConfigs: [] # Custom scrape configurations

In casting.yaml, add this to the deployment collector's document. It scrapes pods annotated signoz.io/scrape: "true" every 60s, at the path in signoz.io/path and the port in signoz.io/port:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              prometheus/scraper:
                config:
                  scrape_configs:
                    - job_name: signoz-scraper
                      scrape_interval: 60s
                      kubernetes_sd_configs:
                        - role: pod
                      relabel_configs:
                        - source_labels: [__meta_kubernetes_pod_annotation_signoz_io_scrape]
                          action: keep
                          regex: "true"
                        - source_labels: [__meta_kubernetes_pod_annotation_signoz_io_path]
                          action: replace
                          regex: (.+)
                          target_label: __metrics_path__
                        - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_signoz_io_port]
                          action: replace
                          separator: ":"
                          target_label: __address__
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, prometheus/scraper]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers otlp/http, otlp/grpc and k8s_cluster.

Custom Scrape Configurations

For more advanced use cases, you can define custom scrape configurations using the scrapeConfigs field. This allows you to specify Prometheus-compatible scrape configs directly:

presets:
  prometheus:
    enabled: true
    scrapeConfigs:
      # Scrape metrics from a specific service
      - job_name: "my-custom-service"
        static_configs:
          - targets: ["my-service.default.svc.cluster.local:9090"]
            labels:
              environment: "production"

In casting.yaml, add a prometheus receiver to the deployment collector's document:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              prometheus:
                config:
                  scrape_configs:
                    - job_name: my-custom-service
                      static_configs:
                        - targets: ["my-service.default.svc.cluster.local:9090"]
                          labels:
                            environment: production
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, prometheus]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers otlp/http, otlp/grpc and k8s_cluster.

Tune Batch and Memory Limiter

Adjust the batch processor for throughput and the memory_limiter processor for stability based on your workload. See Collector Configuration Best Practices for detailed guidance.

The batch processor batches spans, metrics, and logs before export. That reduces connections and improves compression. It supports both size-based and time-based batching.

Configure the batch processor by overriding otelAgent.config.processors.batch and otelDeployment.config.processors.batch in your Helm values. Example override, which raises send_batch_size from the chart default of 10000:

otelAgent:
  config:
    processors:
      batch:
        send_batch_size: 15000
        send_batch_max_size: 20000 # settings can be provided as key value pairs

Any setting supported by the OpenTelemetry batch processor can be set under processors.batch in the chart values.

Foundry's batch processor defaults are send_batch_size: 1000, send_batch_max_size: 2048 and timeout: 10s. In casting.yaml, add this to the agent's document to change them:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              batch:
                send_batch_size: 15000
                send_batch_max_size: 20000

Keys you leave out keep Foundry's values. For the deployment collector, add the same processors.batch block to its document under collector/deployment/deployment.yaml.

Resource Requests and Limits

Recommended resource allocations based on preset combinations:

Configuration	CPU Request	Memory Request	CPU Limit	Memory Limit
Minimal	100m	256Mi	200m	512Mi
Standard	200m	512Mi	500m	1Gi
Full	500m	1Gi	1000m	2Gi

Adjust these values based on your cluster size and monitoring requirements.

Set them under otelAgent.resources and otelDeployment.resources in your Helm values.

In casting.yaml, add a patch to the agent's document. This one sets the Minimal row:

spec:
  patches:
    - target: collectionagent/collector/agent/values.yaml
      operations:
        - op: add
          path: /otelAgent/resources
          value:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi

For the deployment collector, add the same patch to its document with target collectionagent/collector/deployment/values.yaml and path /otelDeployment/resources.

spec:
  patches:
    - target: collectionagent/collector/agent/workload.yaml
      operations:
        - op: replace
          path: /spec/template/spec/containers/0/resources
          value:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi

For the deployment collector, add the same patch to its document with target collectionagent/collector/deployment/workload.yaml.

Foundry's memory_limiter processor uses a fixed limit_mib: 4000. When you set a memory limit, size memory_limiter to it with percentages, in the same document:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              memory_limiter:
                limit_mib: null
                spike_limit_mib: null
                limit_percentage: 80
                spike_limit_percentage: 25

Add Custom Processors to Pipelines

A processor transforms or enriches telemetry once a pipeline lists it under processors.

Defining a processor under otelAgent.config.processors (or otelDeployment.config.processors) adds it to the collector configuration but does not automatically include it in any pipeline. To activate a custom processor, reference it in the relevant pipeline's processors list under otelAgent.config.service.pipelines (or otelDeployment.config.service.pipelines for the deployment collector).

When you override a pipeline's processors list, Helm replaces the entire list, so the chart's defaults for that pipeline are dropped. Always include both your custom processor and every default processor you still want to run in that pipeline.

Foundry's pipelines are traces, metrics and logs, in both collectors. A processor runs once a pipeline lists it. In casting.yaml, add this to the agent's document to define a processor and add it to the traces pipeline:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              attributes/team:
                actions:
                  - key: team
                    value: platform
                    action: insert
            service:
              pipelines:
                traces:
                  processors: [memory_limiter, resourcedetection, resource/identity, k8sattributes, attributes/team, batch]

Lists replace, so restate Foundry's default processors in their order (memory_limiter, resourcedetection, resource/identity, k8sattributes, batch) with yours before batch.

Exposing Ports for Custom Receivers

You may want to use a custom receiver to receive metrics. If the receiver listens on a port, you need to expose that port to the host. The following example shows how to expose port 8125 of statsd receiver:

otelAgent:
  ports:
    statsd:
      enabled: true
      containerPort: 8125
      servicePort: 8125
      hostPort: 8125
      protocol: UDP

In casting.yaml, add the statsd receiver to the agent's document and list it in the metrics pipeline:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              statsd:
                endpoint: 0.0.0.0:8125
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, hostmetrics, kubeletstats, statsd]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers. Then open the port with patches in the same document:

spec:
  patches:
    - target: collectionagent/collector/agent/values.yaml
      operations:
        - op: add
          path: /otelAgent/ports
          value:
            statsd:
              enabled: true
              containerPort: 8125
              servicePort: 8125
              hostPort: 8125
              protocol: UDP

spec:
  patches:
    - target: collectionagent/collector/agent/workload.yaml
      operations:
        - op: add
          path: /spec/template/spec/containers/0/ports/-
          value: {name: statsd, containerPort: 8125, hostPort: 8125, protocol: UDP}
    - target: collectionagent/collector/agent/service.yaml
      operations:
        - op: add
          path: /spec/ports/-
          value: {name: statsd, port: 8125, targetPort: statsd, protocol: UDP}

Collect Metrics from Other Components

The otelDeployment collector is configured to collect cluster metrics and events by default. There are some use cases where you might want to collect metrics from components other than the ones that are being collected by default.

This section describes how to override the default configuration of the otelDeployment collector to collect metrics from additional components. The override-values.yaml file contains the configurations that you can override.

Example 1: Redis Receiver Configuration

The following configuration shows how to configure the otelDeployment collector to collect metrics from a Redis server running inside a namespace "redis-ns" with the name "redis-server" and port 6379:

otelDeployment:
  config:
    receivers:
      redis:
        endpoint: redis-server.redis-ns.svc.cluster.local:6379
    service:
      pipelines:
        metrics/internal:
          receivers: [redis]
          processors: [batch]
          exporters: []

In casting.yaml, add this to the deployment collector's document:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              redis:
                endpoint: redis-server.redis-ns.svc.cluster.local:6379
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, redis]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers otlp/http, otlp/grpc and k8s_cluster.

Example 2: Configuring a PostgreSQL receiver

The following configuration shows how to configure the otelDeployment collector to collect metrics from a PostgreSQL server running inside a namespace "postgres-ns" with the name "postgres-server" and port 5432:

otelDeployment:
  additionalEnvs:
    POSTGRES_USER: <postgres-user>
    POSTGRES_PASSWORD: <postgres-password>
  config:
    receivers:
      postgresql:
        endpoint: postgres-server.postgres-ns.svc.cluster.local:5432
        username: ${env:POSTGRES_USER}
        password: ${env:POSTGRES_PASSWORD}
    service:
      pipelines:
        metrics/internal:
          receivers: [postgresql]
          processors: [batch]
          exporters: []

Verify these values:

<postgres-user>: The PostgreSQL user the receiver logs in as.
<postgres-password>: That user's password. To read it from a Kubernetes Secret instead, see Example 3.

In casting.yaml, add this to the deployment collector's document:

spec:
  collector:
    kind: deployment
    spec:
      env:
        POSTGRES_USER: <postgres-user>
        POSTGRES_PASSWORD: <postgres-password>
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              postgresql:
                endpoint: postgres-server.postgres-ns.svc.cluster.local:5432
                username: ${env:POSTGRES_USER}
                password: ${env:POSTGRES_PASSWORD}
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, postgresql]

Verify these values:

<postgres-user>: The PostgreSQL user the receiver logs in as.
<postgres-password>: That user's password. To read it from a Kubernetes Secret instead, see Example 3.

Lists replace, so the metrics pipeline is restated with Foundry's default receivers otlp/http, otlp/grpc and k8s_cluster.

Example 3: Configuring multiple receivers

The following configuration shows how to configure both Redis and PostgreSQL receivers:

otelDeployment:
  additionalEnvs:
    POSTGRES_USER: <postgres-user>
    POSTGRES_PASSWORD:
      valueFrom:
        secretKeyRef:
          name: postgres
          key: password
  config:
    receivers:
      redis:
        endpoint: redis-server.redis-ns.svc.cluster.local:6379
      postgresql:
        endpoint: postgres-server.postgres-ns.svc.cluster.local:5432
        username: ${env:POSTGRES_USER}
        password: ${env:POSTGRES_PASSWORD}
    service:
      pipelines:
        metrics/internal:
          receivers: [redis, postgresql]
          processors: [batch]
          exporters: []

The password comes from the password key of a Kubernetes Secret named postgres in the collector's namespace.

Verify these values:

<postgres-user>: The PostgreSQL user the receiver logs in as.

In casting.yaml, add this to the deployment collector's document:

spec:
  collector:
    kind: deployment
    spec:
      env:
        POSTGRES_USER: <postgres-user>
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              redis:
                endpoint: redis-server.redis-ns.svc.cluster.local:6379
              postgresql:
                endpoint: postgres-server.postgres-ns.svc.cluster.local:5432
                username: ${env:POSTGRES_USER}
                password: ${env:POSTGRES_PASSWORD}
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, redis, postgresql]

Verify these values:

<postgres-user>: The PostgreSQL user the receiver logs in as.

Then set POSTGRES_PASSWORD from the password key of a Kubernetes Secret named postgres in the collector's namespace, with a patch in the same document:

spec:
  patches:
    - target: collectionagent/collector/deployment/values.yaml
      operations:
        - op: add
          path: /otelDeployment/additionalEnvs/POSTGRES_PASSWORD
          value:
            valueFrom:
              secretKeyRef:
                name: postgres
                key: password

spec:
  patches:
    - target: collectionagent/collector/deployment/workload.yaml
      operations:
        - op: add
          path: /spec/template/spec/containers/0/env/-
          value:
            name: POSTGRES_PASSWORD
            valueFrom:
              secretKeyRef:
                name: postgres
                key: password

The above examples are simple and show how to configure the otelDeployment collector to collect metrics from a single component. However, in a real-world scenario, you might need to configure the receiver with a username and password and other configurations.

Disable Metrics Collection

In case you do not want to collect metrics from your Kubernetes cluster, you can disable using presets in k8s-infra chart.

presets:
  hostMetrics:
    enabled: false
  kubeletMetrics:
    enabled: false
  clusterMetrics:
    enabled: false
 
otelDeployment:
  enabled: false

In casting.yaml, add this to the agent's document to remove the hostmetrics and kubeletstats receivers:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics: null
              kubeletstats: null
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc]

Add this to the deployment collector's document to remove the k8s_cluster receiver:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              k8s_cluster: null
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc]

Lists replace, so each metrics pipeline is restated with Foundry's two OTLP receivers. Kubernetes events keep flowing from the deployment collector.

Common Deployment Scenarios

Production Cluster with Full Monitoring

presets:
  otlphttpExporter:
    enabled: true
  hostMetrics:
    enabled: true
    collectionInterval: 30s
  logsCollection:
    enabled: true
    blacklist:
      enabled: true
      namespaces:
        - kube-system
  kubeletMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
  clusterMetrics:
    enabled: true
  k8sEvents:
    enabled: true

Foundry generates this by default: every preset above, with kube-system logs excluded. In casting.yaml, add this to the agent's document to verify the kubelet's certificate:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              kubeletstats:
                insecure_skip_verify: false

Resource-Constrained Environment

presets:
  otlphttpExporter:
    enabled: true
  hostMetrics:
    enabled: true
    collectionInterval: 60s
  logsCollection:
    enabled: true
    whitelist:
      enabled: true
      namespaces:
        - production
  kubeletMetrics:
    enabled: true
    collectionInterval: 60s
  kubernetesAttributes:
    enabled: true
    passthrough: true

In casting.yaml, add this to the agent's document:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics:
                collection_interval: 60s
              kubeletstats:
                collection_interval: 60s
              filelog/k8s:
                include:
                  - /var/log/pods/production_*/*/*.log
            processors:
              k8sattributes:
                passthrough: true

Lists replace, so include collects logs from the production namespace only.

Development Environment

Telemetry can carry a deployment.environment resource attribute (such as development, staging, or production), set globally or in your application's OTEL_RESOURCE_ATTRIBUTES.

global:
  deploymentEnvironment: <DEPLOYMENT_ENVIRONMENT>

Verify these values:

<DEPLOYMENT_ENVIRONMENT>: Your deployment environment, for example development or staging.

You can specify the environment by including deployment.environment in the OTEL_RESOURCE_ATTRIBUTES environment variable of your application manifest. This will take precedence over the global.deploymentEnvironment setting in the k8s-infra chart. If it is not set, the global value will be used.

env:
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: deployment.environment=<DEPLOYMENT_ENVIRONMENT>

In casting.yaml, set this in both documents, the agent's and the deployment collector's:

spec:
  collector:
    spec:
      env:
        OTEL_RESOURCE_ATTRIBUTES: deployment.environment=<DEPLOYMENT_ENVIRONMENT>

Verify these values:

<DEPLOYMENT_ENVIRONMENT>: Your deployment environment, for example development or staging. Use the same value in both documents.

A deployment.environment that an application sets in its own OTEL_RESOURCE_ATTRIBUTES takes precedence over this value.

Troubleshooting

A custom processor under `otelAgent.config.processors` does not appear to run

Defining a processor in Helm values, or under processors in a Foundry config.data block, renders it into the collector configuration but does not attach it to any pipeline. Confirm the processor is referenced in the relevant pipeline's processors list. Helm values and config.data both replace lists, so restate every processor the pipeline still needs. See Add Custom Processors to Pipelines. After applying the change and restarting the collector pods, verify in SigNoz Explorer that the attribute or transformation introduced by the processor is present on the relevant signal.

Telemetry stopped getting Kubernetes attributes after customizing `podAssociation`

Helm replaces array values rather than merging them, and Foundry's config.data replaces lists the same way. An override on podAssociation (pod_association in config.data) drops the defaults, including the fallbacks the processor needs to associate telemetry with a pod. See the warning under Resource Detection and Kubernetes Attributes for the rules to keep. After restoring the rules and restarting the collector pods, confirm in SigNoz Explorer that resource attributes such as k8s.pod.name and k8s.namespace.name are populated on traces, metrics, or logs from your workloads.

K8s Infra - Configure

Prerequisites

Apply Changes

Send Data from Applications to the Collector

Where and how to send data from your application

Export to SigNoz

Collect Host Metrics

Collect Container Logs

Disable Logs Collection

Collect Kubelet Metrics

Collect Cluster Metrics

Collect Kubernetes Events

Resource Detection and Kubernetes Attributes

Scrape Prometheus Metrics from Pods

Custom Scrape Configurations

Tune Batch and Memory Limiter

Resource Requests and Limits

Add Custom Processors to Pipelines

Exposing Ports for Custom Receivers

Collect Metrics from Other Components

Example 1: Redis Receiver Configuration

Example 2: Configuring a PostgreSQL receiver

Example 3: Configuring multiple receivers

Disable Metrics Collection

Common Deployment Scenarios

Production Cluster with Full Monitoring

Resource-Constrained Environment

Development Environment

Troubleshooting

A custom processor under otelAgent.config.processors does not appear to run

Telemetry stopped getting Kubernetes attributes after customizing podAssociation

Next Steps

A custom processor under `otelAgent.config.processors` does not appear to run

Telemetry stopped getting Kubernetes attributes after customizing `podAssociation`