NGINX OpenTelemetry Instrumentation - Trace Requests

NGINX writes exporter errors to its error log. On a VM, check /var/log/nginx/error.log. In Docker and Kubernetes, run docker logs <container> or kubectl logs deployment/<deployment-name>.

Why does nginx -t report unknown directive "otel_exporter"?

NGINX has not loaded the module. Add load_module modules/ngx_otel_module.so; as the first line of nginx.conf, outside every block, then run nginx -t again. The error can name any otel_* directive.

Why does the log show OTel export failure: Invalid or missing key?

SigNoz rejected the ingestion key. Check the header signoz-ingestion-key value, and confirm that the key belongs to the region in your endpoint. In Docker and Kubernetes, confirm that the SIGNOZ_INGESTION_KEY environment variable reaches the container.

Why does the log show OTel export failure: Stream removed?

The endpoint contains a path, such as /v1/traces. The module exports over OTLP/gRPC, which takes only a host and port. Set the endpoint to https://ingest.<region>.signoz.cloud:443.

Why does the log show OTel export failure: failed to connect to all addresses?

The endpoint lacks the https:// prefix, so the module opens an unencrypted connection to a TLS port. Write the endpoint as https://ingest.<region>.signoz.cloud:443. For a Collector on the same host, check that it listens on port 4317.

Why does the service show as unknown_service:nginx?

The configuration has no otel_service_name directive. Add it to the http context, next to otel_exporter.

Why is the NGINX span in a separate trace from the backend?

By default, NGINX ignores trace headers. It starts a new trace for its own span and forwards the original traceparent header unchanged, so the backend joins the caller's trace instead. Add otel_trace_context propagate; so NGINX continues the incoming trace and passes its own span context upstream.

For more help with missing data, see Debug missing traces, logs, and metrics in SigNoz.

What is the OpenTelemetry Collector?

The Collector sits between NGINX and SigNoz. NGINX sends spans to the Collector, and the Collector forwards them to SigNoz.

Why use it?

• Cleaning up data: Filter out noisy spans, or remove sensitive attributes before they leave your servers.
• Adding context: Tag spans with host, Kubernetes, or cloud metadata.
• One agent for all signals: The same Collector can also scrape NGINX metrics and tail NGINX logs.

To send spans through a Collector, point the exporter at the Collector's OTLP/gRPC receiver and remove the header line, because the Collector holds the ingestion key:

otel_exporter {
    endpoint http://localhost:4317;
}

See Switch from direct export to Collector to set up the Collector, and the Collector configuration guide for its options.

The OpenTelemetry project maintains a separate webserver module for NGINX. Besides the request span, it creates a span for each NGINX module that handles the request, such as ngx_http_rewrite_module. Use it only if you need those per-module spans.

This Dockerfile installs the webserver module on NGINX 1.26.0:

FROM nginx:1.26.0
 
RUN apt-get update && apt-get install -y curl \
 && curl -fsSL https://github.com/open-telemetry/opentelemetry-cpp-contrib/releases/download/webserver%2Fv1.1.0/opentelemetry-webserver-sdk-x64-linux.tgz \
      | tar xz -C /opt \
 && cd /opt/opentelemetry-webserver-sdk && ./install.sh
 
ENV LD_LIBRARY_PATH=/opt/opentelemetry-webserver-sdk/sdk_lib/lib
RUN sed -i '1i load_module /opt/opentelemetry-webserver-sdk/WebServerModule/Nginx/1.26.0/ngx_http_opentelemetry_module.so;' /etc/nginx/nginx.conf
 
COPY opentelemetry_module.conf.template /etc/nginx/templates/opentelemetry_module.conf.template

Create opentelemetry_module.conf.template next to the Dockerfile:

NginxModuleEnabled ON;
NginxModuleOtelSpanExporter otlp;
NginxModuleOtelExporterEndpoint https://ingest.<region>.signoz.cloud:443;
NginxModuleOtelSslEnabled ON;
NginxModuleOtelSslCertificatePath /etc/ssl/certs/ca-certificates.crt;
NginxModuleOtelExporterOtlpHeaders signoz-ingestion-key=${SIGNOZ_INGESTION_KEY};
NginxModuleServiceName <service-name>;
NginxModuleServiceNamespace <service-namespace>;
NginxModuleServiceInstanceId <service-instance-id>;
NginxModuleResolveBackends ON;
NginxModuleTraceAsError ON;

Verify these values:

• <region>: Your SigNoz Cloud region
• <service-name>: A descriptive name for this NGINX instance (for example, edge-nginx)
• <service-namespace>: A namespace for the service (for example, platform)
• <service-instance-id>: A unique ID for this instance (for example, nginx-1)

Build and run it as shown in the Docker tab, passing SIGNOZ_INGESTION_KEY with -e.