Go eBPF Zero-Code Instrumentation with OpenTelemetry

Overview

OpenTelemetry Go automatic instrumentation attaches eBPF probes to a running Go process and exports traces to SigNoz. eBPF is a Linux kernel feature that runs small programs at chosen points in a process. The agent runs beside your application as a separate process. Your binary stays as you built it.

The agent produces traces only. It produces no metrics and no logs. SigNoz derives rate, error, and latency metrics from the spans.

Prerequisites

• A Linux host on amd64 or arm64, with kernel 4.19 or later. On macOS and Windows, run the agent in a Linux container or a virtual machine.
• Permission to run a privileged container as root.
• A Go application that uses at least one supported library.
• The path of the application binary, and the path must be readable by the agent.
• An instance of SigNoz (either Cloud or Self-Hosted).

How it works

The agent runs as its own process next to your application. It reads the target binary, finds the functions it knows how to trace, and attaches uprobes to them. A uprobe is a kernel hook that fires when the process reaches a chosen instruction.

The eBPF programs write events into a kernel ring buffer. The agent reads that buffer, builds spans, and exports them to SigNoz. This is why the agent needs a shared process namespace, root, and a privileged container: it reads another process through /proc and loads programs into the kernel.

Send traces to SigNoz

Step 1. Run the agent next to your application

The agent finds the target process through OTEL_GO_AUTO_TARGET_EXE, which holds the full path of the binary. It shares a process namespace with the application, so it can read that path.

services:
  myapp:
    image: <your-image>
    volumes:
      - ./bin:/app:ro
    command: /app/myapp
 
  go-auto:
    image: otel/autoinstrumentation-go:v0.24.0
    privileged: true
    pid: "host"
    environment:
      OTEL_GO_AUTO_TARGET_EXE: /app/myapp
      OTEL_SERVICE_NAME: "<service-name>"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://ingest.<region>.signoz.cloud:443"
      OTEL_EXPORTER_OTLP_HEADERS: "signoz-ingestion-key=<your-ingestion-key>"
    volumes:
      - ./bin:/app:ro
      - /proc:/host/proc
    depends_on:
      - myapp

docker compose up -d

kubectl create secret generic signoz-ingestion \
  --from-literal=otlp-headers="signoz-ingestion-key=<your-ingestion-key>"

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      # The agent and the application must share a process namespace.
      shareProcessNamespace: true
      containers:
        - name: myapp
          image: <your-image>
          ports:
            - containerPort: 8080
 
        - name: autoinstrumentation-go
          image: otel/autoinstrumentation-go:v0.24.0
          imagePullPolicy: IfNotPresent
          env:
            - name: OTEL_GO_AUTO_TARGET_EXE
              value: /usr/local/bin/myapp
            - name: OTEL_SERVICE_NAME
              value: "<service-name>"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "https://ingest.<region>.signoz.cloud:443"
            - name: OTEL_EXPORTER_OTLP_HEADERS
              valueFrom:
                secretKeyRef:
                  name: signoz-ingestion
                  key: otlp-headers
          securityContext:
            runAsUser: 0
            privileged: true

kubectl apply -f deployment.yaml

kubectl label namespace <your-namespace> pod-security.kubernetes.io/enforce=privileged

docker run --rm \
  --privileged \
  --pid=host \
  -v /usr/local/bin/myapp:/usr/local/bin/myapp:ro \
  -v /proc:/host/proc \
  -e OTEL_GO_AUTO_TARGET_EXE=/usr/local/bin/myapp \
  -e OTEL_SERVICE_NAME="<service-name>" \
  -e OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.signoz.cloud:443" \
  -e OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>" \
  otel/autoinstrumentation-go:v0.24.0

Verify these values:

• <region>: Your SigNoz Cloud region.
• <your-ingestion-key>: Your SigNoz ingestion key.
• <service-name>: The name your service gets in SigNoz, for example payment-service.
• OTEL_GO_AUTO_TARGET_EXE: The full path of the application binary, not the process name.

If the application is not running yet, the agent waits for it to start.

Step 2. Confirm that the agent attached

Read the agent logs. A successful start ends with these lines:

loading probe
instrumentation loaded successfully, starting...

The agent also logs Offset not cached, analyzing directly for each field it reads from the binary. You see this when your Go version or library version is newer than the agent release. The agent then reads the offsets from the DWARF data in the binary.

This fallback needs debug information. A binary built with -ldflags "-s -w" carries none, so the agent fails to load the probe instead of continuing. Either build that binary without stripping, or run an agent release whose cache covers your versions.

Validate

Send a few requests to your application, then open SigNoz.

Open Services. Your service appears with request rate, error rate, and latency within a minute or two.

Open Traces and filter by your service name. HTTP server spans use the route as the name, for example GET /rolldice.

Open one trace. Database calls and outbound HTTP calls appear as child spans of the request.

Supported libraries

The agent traces four libraries:

The agent takes no new probes until the new probe API is ready, so expect few additions. For the version ranges of the release this guide pins, see the compatibility matrix for v0.24.0.

OTEL_GO_AUTO_INCLUDE_DB_STATEMENT=true
OTEL_GO_AUTO_PARSE_DB_STATEMENT=true

OTEL_GO_AUTO_GLOBAL=true

OTEL_EXPORTER_OTLP_ENDPOINT="http://<collector-host>:4318"

Limitations

• The agent runs on Linux only. On macOS and Windows, it must run in a Linux container or a virtual machine.
• The agent needs a privileged container and the root user.
• It traces four libraries. Your own code and other frameworks get no spans.
• It produces traces only, so you get no Go runtime metrics.
• The OpenTelemetry project still labels the agent as work in progress. Test it on a non-production service first.

Next steps

• Compare this page with Go compile-time instrumentation, which is stable and covers more libraries.
• Use OBI when you want kernel-level metrics and traces for services in any language.
• Set up alerts on error rate and latency.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.