OpenTelemetry Operator - Configure

The install guide provides a working config: the agent collects host and kubelet metrics, container logs, and data from instrumented applications on every node, and the cluster collector collects cluster metrics and Kubernetes events. This page explains what each component in that config does and how to customize it through Foundry's casting.yaml.

Prerequisites

• Collection Agent installed with Foundry (see Installation guide)
• SigNoz Cloud account or self-hosted SigNoz instance
• Access to the cluster with kubectl, and the casting.yaml and foundryctl you installed with

Apply Changes

For each change:

1. Add the section's snippet to casting.yaml, in the agent's document or in the cluster collector's document (collector.kind: deployment). Foundry already generates the default receivers and processors, so add only what you change.

2. Run this from the directory that contains casting.yaml to apply the change:

   Copy

   foundryctl cast -f casting.yaml

   Foundry updates the collector resources, and the operator rolls the collector pods with the new config.

3. Run this on the machine where you use kubectl to check that the pods restart:

   Copy

   kubectl get pods -n signoz

   Then view the collector logs to confirm there are no configuration errors:

   Copy

   kubectl logs -n signoz daemonset/signoz-collector-agent-collector --since=5m

   For the cluster collector, use deployment/signoz-collector-deployment-collector.

4. In SigNoz, confirm data in Infrastructure Monitoring for node, pod, and container metrics, Logs for container logs and Kubernetes events, and Traces for application traces.

Send Data from Applications to the Collector

Node, pod, and container metrics, container logs, and Kubernetes events flow as soon as the collector pods are running.

Where and how to send data from your application

To send traces, metrics, and logs from an application, instrument it with OpenTelemetry (language guides), then add this to the application's container in its manifest:

env:
  - name: K8S_POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: K8S_POD_UID
    valueFrom:
      fieldRef:
        fieldPath: metadata.uid
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://signoz-collector-agent-collector.signoz.svc.cluster.local:4318
  - name: OTEL_SERVICE_NAME
    value: <service-name>
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.pod.ip=$(K8S_POD_IP),k8s.pod.uid=$(K8S_POD_UID)

Verify these values:

• <service-name>: The name your application shows under in SigNoz.
• signoz in the address: metadata.name, which is also the namespace unless the foundry.signoz.io/kubernetes-namespace annotation sets another.

The Service keeps each application's traffic on the agent of its own node, which adds the Kubernetes resource attributes, matched by k8s.pod.ip and k8s.pod.uid. Use port 4317 for OTLP over gRPC.

To have the operator inject this instead, see Auto-instrumentation.

Redeploy the application and send it some traffic. Within a minute it shows under Services in SigNoz, and its traces carry k8s.namespace.name and k8s.pod.name.

Export to SigNoz

The OTLP/HTTP exporter sends the telemetry that both collectors gather to SigNoz.

Foundry generates this by default: the otlphttp/signoz exporter sends every pipeline to SIGNOZ_INGESTION_ENDPOINT.

Collect Host Metrics

Use the hostmetrics receiver to scrape CPU, memory, disk, filesystem, load, and network metrics from each node.

Foundry collects host metrics by default, every 30s. In casting.yaml, add this to the agent's document to change the interval and the excluded disk devices:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics:
                collection_interval: 60s
                scrapers:
                  disk:
                    exclude:
                      devices: [^ram\d+$, ^loop\d+$]

Lists replace, so devices holds only the entries you list. Foundry's default list is ^ram\d+$, ^zram\d+$, ^loop\d+$, ^fd\d+$, ^hd[a-z]\d+$, ^sd[a-z]\d+$, ^vd[a-z]\d+$, ^xvd[a-z]\d+$ and ^nvme\d+n\d+p\d+$; restate the ones you keep. The filesystem and network exclusion lists work the same way.

Collect Container Logs

Use the filelog receiver to tail the container log files on each node. For full setup details, see Collecting Kubernetes Pod Logs.

Foundry collects container logs by default, from the end of each file, without the collectors' own logs and kube-system. In casting.yaml, add this to the agent's document to read from the beginning and exclude more pods:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/k8s:
                start_at: beginning
                exclude:
                  - /var/log/pods/signoz_signoz-collector-*/*/*.log
                  - /var/log/pods/kube-system_*/*/*.log
                  - /var/log/pods/*_hotrod*_*/*/*.log
                  - /var/log/pods/*_locust*_*/*/*.log

Lists replace, so restate Foundry's two default exclusions (the first two entries) before your own. The first one skips the collectors' own logs and follows <namespace>_<metadata.name>-collector-*. Exclude a namespace with /var/log/pods/<namespace>_*/*/*.log, a pod with /var/log/pods/*_<pod>*_*/*/*.log, or a container with /var/log/pods/*_*_*/<container>/*.log.

To collect logs from some namespaces only, replace include:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/k8s:
                include:
                  - /var/log/pods/production_*/*/*.log

Disable Logs Collection

In casting.yaml, add this to the agent's document. It removes the filelog/k8s receiver; logs that applications send over OTLP keep flowing:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              filelog/k8s: null
            service:
              pipelines:
                logs:
                  receivers: [otlp/http, otlp/grpc]

null removes the receiver. Lists replace, so the logs pipeline is restated with Foundry's two OTLP receivers.

Collect Kubelet Metrics

Use the kubeletstats receiver to collect node, pod, container, and volume metrics from the kubelet on each node.

Foundry collects kubelet metrics by default, every 30s, with the metrics in the table below enabled. In casting.yaml, add this to the agent's document to change the interval, verify the kubelet's certificate, and enable one more metric:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              kubeletstats:
                collection_interval: 60s
                insecure_skip_verify: false
                metrics:
                  k8s.pod.memory.node.utilization:
                    enabled: true

metrics is a map, so the metrics you add join Foundry's defaults. With insecure_skip_verify: false, the receiver checks the kubelet's certificate against the cluster CA.

These container-level metrics are disabled by default in the upstream kubeletstats receiver but enabled by default by Foundry. Keep them enabled to retain full container visibility on the Containers view:

Collect Cluster Metrics

Use the k8s_cluster receiver on the cluster collector to collect cluster-level metrics from the Kubernetes API server.

Foundry collects cluster metrics by default from the cluster collector, every 30s, with the metrics in the table below enabled. In casting.yaml, add this to the cluster collector's document to change the interval and report storage too:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              k8s_cluster:
                collection_interval: 60s
                allocatable_types_to_report: [cpu, memory, storage]

Lists replace, so restate every entry you keep. Foundry's defaults are allocatable_types_to_report: [cpu, memory] and node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable].

These metrics are disabled by default in the upstream k8s_cluster receiver but enabled by default by Foundry. Keep them enabled to retain pod- and container-status visibility. Without them, status falls back to the lifecycle phase or state only:

Collect Kubernetes Events

Use the k8s_events receiver on the cluster collector to collect Kubernetes events as logs.

Foundry collects Kubernetes events by default from the cluster collector, for all namespaces. In casting.yaml, add this to the cluster collector's document to watch specific namespaces:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              k8s_events:
                namespaces: [default, production]

Resource Detection and Kubernetes Attributes

The resourcedetection processor adds host and cloud resource attributes, and the k8sattributes processor adds Kubernetes resource attributes, so you can filter and group by namespace, workload, or pod in SigNoz.

Foundry enriches telemetry by default with the k8sattributes processor, with the default extract.metadata attributes and pod association rules. In casting.yaml, add this to the agent's document to extract a pod label and a node annotation:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              k8sattributes:
                extract:
                  labels:
                    - tag_name: service.name
                      key: app.kubernetes.io/name
                      from: pod
                  annotations:
                    - tag_name: node.ttl
                      key: node.alpha.kubernetes.io/ttl
                      from: node

Lists replace, so restating extract.metadata or pod_association drops Foundry's defaults: include every entry you keep, as the warning below describes. Set passthrough: true under k8sattributes to skip Kubernetes API lookups.

Scrape Prometheus Metrics from Pods

The cluster collector can scrape metrics from pods that expose a Prometheus endpoint. In casting.yaml, add this to the cluster collector's document. It scrapes pods annotated signoz.io/scrape: "true" every 60s, at the path in signoz.io/path and the port in signoz.io/port:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              prometheus/scraper:
                config:
                  scrape_configs:
                    - job_name: signoz-scraper
                      scrape_interval: 60s
                      kubernetes_sd_configs:
                        - role: pod
                      relabel_configs:
                        - source_labels: [__meta_kubernetes_pod_annotation_signoz_io_scrape]
                          action: keep
                          regex: "true"
                        - source_labels: [__meta_kubernetes_pod_annotation_signoz_io_path]
                          action: replace
                          regex: (.+)
                          target_label: __metrics_path__
                        - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_signoz_io_port]
                          action: replace
                          separator: ":"
                          target_label: __address__
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, prometheus/scraper]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers otlp/http, otlp/grpc and k8s_cluster.

Custom Scrape Configurations

For more advanced use cases, add a prometheus receiver with your own scrape configs to the cluster collector's document:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              prometheus:
                config:
                  scrape_configs:
                    - job_name: my-custom-service
                      static_configs:
                        - targets: ["my-service.default.svc.cluster.local:9090"]
                          labels:
                            environment: production
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, k8s_cluster, prometheus]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers otlp/http, otlp/grpc and k8s_cluster.

Tune Batch and Memory Limiter

Adjust the batch processor for throughput and the memory_limiter processor for stability based on your workload. See Collector Configuration Best Practices for detailed guidance.

The batch processor batches spans, metrics, and logs before export. That reduces connections and improves compression. It supports both size-based and time-based batching.

Foundry's batch processor defaults are send_batch_size: 1000, send_batch_max_size: 2048 and timeout: 10s. In casting.yaml, add this to the agent's document to change them:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              batch:
                send_batch_size: 15000
                send_batch_max_size: 20000

Keys you leave out keep Foundry's values. For the cluster collector, add the same processors.batch block to its document under collector/deployment/deployment.yaml.

Resource Requests and Limits

Recommended resource allocations:

Adjust these values based on your cluster size and monitoring requirements. In casting.yaml, add a patch to the agent's document. This one sets the Minimal row:

spec:
  patches:
    - target: collectionagent/collector/agent/values.yaml
      operations:
        - op: add
          path: /collectors/agent/resources
          value:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi

spec:
  patches:
    - target: collectionagent/collector/agent/opentelemetrycollector.yaml
      operations:
        - op: replace
          path: /spec/resources
          value:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 200m
              memory: 512Mi

Foundry's memory_limiter processor uses a fixed limit_mib: 4000. When you set a memory limit, size memory_limiter to it with percentages, in the same document:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              memory_limiter:
                limit_mib: null
                spike_limit_mib: null
                limit_percentage: 80
                spike_limit_percentage: 25

Add Custom Processors to Pipelines

A processor transforms or enriches telemetry once a pipeline lists it under processors.

Foundry's pipelines are traces, metrics and logs, in both collectors. A processor runs once a pipeline lists it. In casting.yaml, add this to the agent's document to define a processor and add it to the traces pipeline:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            processors:
              attributes/team:
                actions:
                  - key: team
                    value: platform
                    action: insert
            service:
              pipelines:
                traces:
                  processors: [memory_limiter, resourcedetection, resource/identity, k8sattributes, attributes/team, batch]

Lists replace, so restate Foundry's default processors in their order (memory_limiter, resourcedetection, resource/identity, k8sattributes, batch) with yours before batch.

Exposing Ports for Custom Receivers

A receiver that listens on a port needs that port open on the agent. The following example opens port 8125 for the statsd receiver. In casting.yaml, add the statsd receiver to the agent's document and list it in the metrics pipeline:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              statsd:
                endpoint: 0.0.0.0:8125
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc, hostmetrics, kubeletstats, statsd]

Lists replace, so the metrics pipeline is restated with Foundry's default receivers. Then open the port on the node and on the agent's Service with a patch in the same document:

spec:
  patches:
    - target: collectionagent/collector/agent/values.yaml
      operations:
        - op: add
          path: /collectors/agent/ports/-
          value: {name: statsd, port: 8125, hostPort: 8125, protocol: UDP}

spec:
  patches:
    - target: collectionagent/collector/agent/opentelemetrycollector.yaml
      operations:
        - op: add
          path: /spec/ports/-
          value: {name: statsd, port: 8125, hostPort: 8125, protocol: UDP}

Disable Metrics Collection

In casting.yaml, add this to the agent's document to remove the hostmetrics and kubeletstats receivers:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics: null
              kubeletstats: null
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc]

Add this to the cluster collector's document to remove the k8s_cluster receiver:

spec:
  collector:
    kind: deployment
    spec:
      config:
        data:
          collector/deployment/deployment.yaml: |
            receivers:
              k8s_cluster: null
            service:
              pipelines:
                metrics:
                  receivers: [otlp/http, otlp/grpc]

Lists replace, so each metrics pipeline is restated with Foundry's two OTLP receivers. Kubernetes events keep flowing from the cluster collector.

Common Deployment Scenarios

Production Cluster with Full Monitoring

Foundry generates full monitoring by default: host, kubelet, and cluster metrics, container logs without kube-system, Kubernetes attributes, and Kubernetes events. In casting.yaml, add this to the agent's document to verify the kubelet's certificate:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              kubeletstats:
                insecure_skip_verify: false

Resource-Constrained Environment

In casting.yaml, add this to the agent's document:

spec:
  collector:
    spec:
      config:
        data:
          collector/agent/agent.yaml: |
            receivers:
              hostmetrics:
                collection_interval: 60s
              kubeletstats:
                collection_interval: 60s
              filelog/k8s:
                include:
                  - /var/log/pods/production_*/*/*.log
            processors:
              k8sattributes:
                passthrough: true

Lists replace, so include collects logs from the production namespace only.

Development Environment

Telemetry can carry a deployment.environment resource attribute (such as development, staging, or production), set globally or in your application's OTEL_RESOURCE_ATTRIBUTES.

In casting.yaml, set this in both documents, the agent's and the cluster collector's:

spec:
  collector:
    spec:
      env:
        OTEL_RESOURCE_ATTRIBUTES: deployment.environment=<DEPLOYMENT_ENVIRONMENT>

Verify these values:

• <DEPLOYMENT_ENVIRONMENT>: Your deployment environment, for example development or staging. Use the same value in both documents.

A deployment.environment that an application sets in its own OTEL_RESOURCE_ATTRIBUTES takes precedence over this value.

Auto-instrumentation with OpenTelemetry Operator

The operator can also inject OpenTelemetry auto-instrumentation into your workloads with an Instrumentation resource and pod annotations, which produces traces, metrics, and logs without code changes. In the Instrumentation resource, set the exporter endpoint to the agent's Service:

spec:
  exporter:
    endpoint: http://signoz-collector-agent-collector.signoz.svc.cluster.local:4318

If you changed metadata.name or the namespace, find the Service name in Send Data from Applications to the Collector.

• For the Instrumentation resource and the annotations, see the upstream guide: Automatic instrumentation via the Operator.
• For language-specific steps, use the SigNoz instrumentation docs. For example, Java: OTel Operator steps.

Next Steps

• OpenTelemetry Operator Overview
• Telemetry Data Requirements for Infrastructure Monitoring
• Collector Configuration Best Practices