K8s Serverless (EKS Fargate) - OpenTelemetry Operator

This guide runs the EKS Fargate Collection Agent through the OpenTelemetry Operator. Foundry generates each collector as an OpenTelemetryCollector resource, and the operator creates the deployment collector's workload and Service from it. For the sidecar, the operator's webhook injects the collector into each new pod whose template carries the sidecar.opentelemetry.io/inject annotation.

Prerequisites

• An EKS cluster with a Fargate profile that selects the collector's namespace and your applications' namespaces.
• kubectl configured to reach the cluster, with the AWS CLI and Helm 3 installed on the same machine for the Fargate profile and cert-manager.
• A SigNoz to send data to: SigNoz Cloud or a self-hosted SigNoz.

Install the Collection Agent

Step 1: Install the OpenTelemetry Operator

Install the OpenTelemetry Operator, which reconciles the OpenTelemetryCollector resources Foundry generates for the collectors, and cert-manager, which issues the certificate of the operator's admission webhook. Run these on the machine where you use kubectl, with the AWS CLI and Helm 3 installed:

1. Add a Fargate profile that selects the cert-manager and opentelemetry-operator-system namespaces:

   Copy

   aws eks create-fargate-profile --cluster-name <cluster-name> \
     --fargate-profile-name opentelemetry-operator \
     --pod-execution-role-arn <pod-execution-role-arn> \
     --selectors namespace=cert-manager namespace=opentelemetry-operator-system

   Verify these values:

   • <cluster-name>: Your EKS cluster's name.
   • <pod-execution-role-arn>: The ARN of the pod execution role your other Fargate profiles use.

2. Install cert-manager with its webhook on port 10260, since on Fargate the kubelet holds the default port 10250, as cert-manager's AWS Fargate notes describe:

   Copy

   helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
     --version v1.21.2 \
     --namespace cert-manager \
     --create-namespace \
     --set crds.enabled=true \
     --set webhook.securePort=10260

3. Wait until cert-manager's webhook is ready, so that it accepts the operator's certificate:

   Copy

   kubectl -n cert-manager rollout status deploy/cert-manager-webhook

4. Install operator v0.139.0, which matches the OpenTelemetry Collector 0.139.0 that Foundry pins, and its CRDs:

   Copy

   kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/download/v0.139.0/opentelemetry-operator.yaml

5. Check the operator. It is ready when READY shows 1/1:

   Copy

   kubectl get deployment opentelemetry-operator-controller-manager -n opentelemetry-operator-system

Step 2: Install foundryctl

Run this on the machine where you use kubectl:

curl -fsSL https://signoz.io/foundry.sh | bash

For manual install (Windows PowerShell, air-gapped, etc.) or PATH setup, see the Foundry getting-started guide.

Step 3: Create casting.yaml

Create a file named casting.yaml with one document for the deployment collector described in How it works. collector.kind: deployment selects it:

apiVersion: v1alpha1
kind: CollectionAgent
metadata:
  name: signoz
  annotations:
    foundry.signoz.io/kubernetes-collector-controller: opentelemetry-operator
spec:
  deployment:
    platform: aws
    mode: kubernetes-serverless
    flavor: kustomize
  collector:
    kind: deployment
    spec:
      env:
        SIGNOZ_INGESTION_ENDPOINT: https://ingest.<region>.signoz.cloud:443
        SIGNOZ_INGESTION_KEY: <your-ingestion-key>
        K8S_CLUSTER_NAME: <CLUSTER_NAME>
        OTEL_RESOURCE_ATTRIBUTES: deployment.environment=<DEPLOYMENT_ENVIRONMENT>
      config:
        data:
          collector/deployment/deployment.yaml: |
            exporters:
              otlphttp/signoz:
                headers:
                  signoz-ingestion-key: ${env:SIGNOZ_INGESTION_KEY}

Verify these values:

• <region>: Your SigNoz Cloud region.
• <your-ingestion-key>: Your SigNoz Cloud ingestion key.
• <CLUSTER_NAME>: A name for this cluster, unique across the clusters that send to the same SigNoz, which becomes k8s.cluster.name and names the cluster in the SigNoz Kubernetes views.
• <DEPLOYMENT_ENVIRONMENT>: Your deployment environment, for example staging or production, which becomes deployment.environment on the collected telemetry.

For every option of this casting, see the EKS Fargate Kustomize example.

metadata:
  name: signoz
  annotations:
    foundry.signoz.io/kubernetes-collector-controller: opentelemetry-operator
    foundry.signoz.io/kubernetes-namespace: signoz   # namespace to apply into, created by the apply

Step 4: Deploy

Run this from the directory that contains casting.yaml:

foundryctl cast -f casting.yaml

This command:

• writes the kustomize root pours/collectionagent/collector/deployment;
• applies it with kubectl apply -k to the cluster of your current kubectl context;
• creates the collector's namespace on the first apply, signoz from metadata.name in this casting. The root holds the namespace, the collector's service account and RBAC, and the OpenTelemetryCollector resource signoz-collector-deployment.

The operator then creates the collector's workload and Service, signoz-collector-deployment-collector, from the resource. Run the same command again after editing casting.yaml to apply the change.

foundryctl forge -f casting.yaml   # writes the kustomize root under pours/collectionagent/collector/deployment
kubectl apply -k pours/collectionagent/collector/deployment

Step 5: Verify the Collection Agent

Run this on the machine where you use kubectl to check that the deployment collector's pod is running on a Fargate node, in the namespace from metadata.name:

kubectl get pods -n signoz -o wide

The output should look similar to the following, with one signoz-collector-deployment-collector pod on a node whose name starts with fargate-. The operator names the workload after its OpenTelemetryCollector resource with a -collector suffix. Pod suffixes, IP addresses, and node names vary:

NAME                                                     READY   STATUS    RESTARTS   AGE   IP           NODE                                 NOMINATED NODE   READINESS GATES
signoz-collector-deployment-collector-6d9c7b8f5c-k2x9p   1/1     Running   0          2m    10.0.12.34   fargate-ip-10-0-12-34.ec2.internal   <none>           <none>

In SigNoz, open the Kubernetes Monitoring views under Infrastructure Monitoring. The Clusters view lists your cluster under the name you set as <CLUSTER_NAME>, and the Pods and Nodes views fill in as data arrives. Check Logs Explorer to confirm Kubernetes events are flowing.

Add the sidecar to your workloads

The sidecar has its own casting.yaml, kept in your application's kustomization directory.

Step 1: Create casting.yaml

The sidecar's document carries the same foundry.signoz.io/kubernetes-collector-controller: opentelemetry-operator annotation as the deployment collector's.

Create a file named casting.yaml with this content:

apiVersion: v1alpha1
kind: CollectionAgent
metadata:
  name: signoz
  annotations:
    foundry.signoz.io/kubernetes-collector-controller: opentelemetry-operator
spec:
  deployment:
    platform: aws
    mode: kubernetes-serverless
    flavor: kustomize
  collector:
    kind: sidecar
    spec:
      env:
        SIGNOZ_INGESTION_ENDPOINT: https://ingest.<region>.signoz.cloud:443
        SIGNOZ_INGESTION_KEY: <your-ingestion-key>
        K8S_CLUSTER_NAME: <CLUSTER_NAME>
        OTEL_RESOURCE_ATTRIBUTES: deployment.environment=<DEPLOYMENT_ENVIRONMENT>
      config:
        data:
          collector/sidecar/sidecar.yaml: |
            exporters:
              otlphttp/signoz:
                headers:
                  signoz-ingestion-key: ${env:SIGNOZ_INGESTION_KEY}

Verify these values:

• <region>: Your SigNoz Cloud region.
• <your-ingestion-key>: Your SigNoz Cloud ingestion key.
• <CLUSTER_NAME>: The same value as in the deployment collector's casting.yaml, which becomes k8s.cluster.name on every signal.
• <DEPLOYMENT_ENVIRONMENT>: Your deployment environment, for example staging or production, which becomes deployment.environment on the collected telemetry.

Step 2: Generate the sidecar files

Keep casting.yaml in your application's kustomization directory, the one that holds its kustomization.yaml, so pours/ lands beside kustomization.yaml. Run this there to write the sidecar's files:

foundryctl forge -f casting.yaml

Foundry generates the OpenTelemetryCollector resource signoz-collector-sidecar in sidecar mode. The operator injects the sidecar into each new pod whose template carries the sidecar.opentelemetry.io/inject annotation.

Step 3: Add the sidecar to your workloads

Add the sidecar's files to your application's kustomization.yaml, then annotate each workload that takes the sidecar.

The sidecar's permissions go to the namespace's default service account. If your workloads run under service accounts of their own, create signoz-rolebinding.yaml beside kustomization.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: signoz-collector-sidecar
subjects:
  - kind: ServiceAccount
    name: <your-service-account>
  - kind: ServiceAccount
    name: default

Verify these values:

• <your-service-account>: The service account of a workload that takes the sidecar, from serviceAccountName in its pod template; add one subject per account, and keep default while any such workload runs under it.

Edit your application's kustomization.yaml to add the sidecar's directory and the rolebinding file:

 namespace: <your-namespace>
 resources:
+  - pours/collectionagent/collector/sidecar
 patches:
+  - path: signoz-rolebinding.yaml

Verify these values:

• <your-namespace>: Your application's namespace.

Leave out the signoz-rolebinding.yaml entry if you did not create that file. The kustomization's namespace: places the sidecar's resource and permissions in your application's namespace.

Add the annotation to the pod template of each workload that takes the sidecar, whichever way you manage the template:

 spec:
   template:
     metadata:
+      annotations:
+        sidecar.opentelemetry.io/inject: "signoz-collector-sidecar"

A CronJob takes the annotation under spec.jobTemplate.spec.template.metadata.annotations.

The sidecar's resource must exist before your workloads' pods are created. On the first deploy, apply the sidecar's files on their own and check that the resource is listed. Run this from the directory that contains casting.yaml, with the same <your-namespace>:

kubectl -n <your-namespace> apply -k pours/collectionagent/collector/sidecar
kubectl -n <your-namespace> get opentelemetrycollector signoz-collector-sidecar

Then run this on the machine where you use kubectl to apply your application:

kubectl apply -k <overlay>

Verify these values:

• <overlay>: The directory that holds your application's kustomization.yaml, relative to where you run the command.

The operator injects the sidecar into each new pod, so restart the workloads whose pods were already running, for example a Deployment:

kubectl -n <your-namespace> rollout restart deployment/<your-workload>

Verify these values:

• <your-workload>: The name of the workload, from its manifest.

Jobs and CronJob runs created before the resource ran without the sidecar; delete them and create them again.

Step 4: Verify the sidecar

Run this on the machine where you use kubectl to check that a workload's pods run the sidecar:

kubectl -n <your-namespace> get pods -l <your-pod-selector> -o jsonpath='{.items[*].spec.initContainers[*].name}'

Verify these values:

• <your-namespace>: Your application's namespace.
• <your-pod-selector>: A label selector for the workload's pods, from spec.selector.matchLabels in its manifest.

The output lists otc-container, the sidecar the operator injected, once per pod.

Once your application sends to the sidecar, as Send Data from Applications to the Collector shows, its traces, metrics, and logs appear in SigNoz with the k8s.pod.name and k8s.namespace.name resource attributes.

Apply Changes

After a change to the sidecar's casting.yaml, forge again, apply your application again, and restart the workloads that take the sidecar.

Customize the Collection Agent

Every change follows the same loop: edit casting.yaml, then run foundryctl cast -f casting.yaml again. For the sidecar, forge and apply as Apply Changes shows, then restart the workloads that run it.

• Collector settings (image, environment): set them in the collector's spec (see spec fields). For example, to pin the collector image:

  Copy

  spec:
    collector:
      spec:
        image: otel/opentelemetry-collector-contrib:0.139.0

• Collector config (receivers, processors, pipelines): add it under spec.collector.spec.config.data, keyed collector/deployment/deployment.yaml in the deployment collector's document or collector/sidecar/sidecar.yaml in the sidecar's. It merges into the generated config, which becomes the resource's spec.config (see custom config files). For example, to scrape kubelet metrics every 60s:

  Copy

  spec:
    collector:
      spec:
        config:
          data:
            collector/deployment/deployment.yaml: |
              receivers:
                receiver_creator:
                  receivers:
                    kubeletstats:
                      config:
                        collection_interval: 60s

• Manifests (resources, annotations): add patches under spec.patches against collectionagent/collector/deployment/opentelemetrycollector.yaml or collectionagent/collector/sidecar/opentelemetrycollector.yaml, the collectors' OpenTelemetryCollector resources, with paths under /spec. For example, to give the deployment collector more CPU and memory:

  Copy

  spec:
    patches:
      - target: collectionagent/collector/deployment/opentelemetrycollector.yaml
        operations:
          - op: replace
            path: /spec/resources
            value:
              requests:
                cpu: 500m
                memory: 1Gi
              limits:
                cpu: 500m
                memory: 1Gi

  For every field of the resource, see the OpenTelemetryCollector API reference. Each run regenerates the files under pours/, so keep every change in casting.yaml.

kubectl describe pod -n signoz <pod-name>

kubectl rollout restart -n signoz deployment/signoz-collector-deployment-collector

kubectl logs -n signoz deployment/signoz-collector-deployment-collector --since=10m

kubectl auth can-i get nodes/proxy --as=system:serviceaccount:signoz:signoz-collector-deployment

foundryctl forge -f casting.yaml

kubectl config current-context

Next Steps

• Configure the EKS Fargate Collection Agent
• Learn more about the EKS Fargate Collection Agent
• Learn more about the OpenTelemetry Operator
• Telemetry Data Requirements for Infrastructure Monitoring