Monitor EMQX with OpenTelemetry Metrics, Traces, Logs

Metrics

EMQX 6.3.0 exports 142 metrics. Names are dotted and carry no emqx prefix, so scope every query by service.name to avoid mixing them with metrics from other sources.

Some gauges are cluster-wide and arrive with the same value from every node: topics.count, routes.count, retained.count, subscriptions.shared.count, durable_subscriptions.count, cluster_sessions.count, and node.running. Take the maximum across nodes for these, not the sum. Counts such as connections.count, sessions.count, and subscriptions.count are node-local, so sum those.

Spans

Logs

Log records carry the same resource attributes as the other two signals, plus Erlang context such as pid, file, line, and mfa. Bodies are strings for plain messages and structured maps for EMQX's own event reports.

Nothing arrives in SigNoz

• Read the broker log for [grpc_client] connect to ... failed or opentelemetry_exporter failed to export.
• Make sure that the endpoint scheme matches your intent. EMQX turns TLS on by itself for an https:// endpoint, but it does not verify the server certificate unless you set verify = verify_peer.
• Make sure that the ingestion key header name is quoted. Without quotes, HOCON reads signoz-ingestion-key as a nested path and the header breaks.
• Make sure that the node can reach the endpoint: curl -v https://ingest.<region>.signoz.cloud:443.
• Make sure that at least one signal is enabled. All three default to enable = false.

TLS handshake fails after you enable verification

The log shows failed to export with {tls_alert,{unknown_ca,...}}. EMQX cannot build a trust chain to the SigNoz certificate.

• Make sure that cacertfile points at your system trust store, not at EMQX's bundled cacert.pem. The bundled file is a demo CA and verifies nothing public.
• Make sure that the path exists inside the container or host that runs EMQX.
• Leave server_name_indication unset. A value that does not match the endpoint host fails the same way.

EMQX rejects the opentelemetry configuration

The build has no exporter. Run emqx ctl status and compare the version against the table in Prerequisites. Open source builds from 5.8.3 to 5.8.9 exclude the exporter.

Collect metrics through the Prometheus endpoint instead. On these builds prometheus.enable_basic_auth defaults to false, so the endpoint answers unauthenticated scrapes. Add this receiver to your OpenTelemetry Collector configuration. Reference it from a metrics pipeline:

receivers:
  prometheus/emqx:
    config:
      scrape_configs:
        - job_name: emqx
          scrape_interval: 30s
          metrics_path: /api/v5/prometheus/stats
          static_configs:
            - targets: ["<emqx-host>:18083"]

EMQX names these metrics emqx_*, so they do not match the names in Data Collected. This path carries no traces and no logs.

Port 18083 also serves the EMQX Dashboard, so keep it off the public network and reachable only from the Collector.

No logs appear

opentelemetry.logs.level cannot be more verbose than the broker's own handlers. If log.console.level is warning, the broker exports only warnings and above, even when you set the OpenTelemetry level to info. Raise both levels, or leave the default.

Keep logs.level = warning in production. At info, EMQX exports Erlang supervisor progress reports, which are large and constant.

Filtering on severity returns nothing

severity_text holds values such as SEVERITY_NUMBER_WARN, not WARN. Filter on severity_number, or copy the exact value from a record in the Logs Explorer.

Some published messages produce no spans

Trace context travels in MQTT user properties, which exist only in MQTT 5.0. A 3.1.1 publisher sends no traceparent, so EMQX skips the message. Set traces.filter.trace_all = true to trace it anyway. EMQX then generates a fresh trace ID.

Metric names collide with other services

cpu.use, run.queue, used.memory, and similar names carry no vendor prefix. Filter by service.name = <service-name>, or rename them in a Collector with the metricstransform processor.

A distinct OTEL_SERVICE_NAME per cluster keeps two EMQX deployments apart in the same workspace.

Send EMQX telemetry to a Collector first for any of these reasons:

• You run EMQX 5.x, whose exporter cannot attach the SigNoz ingestion key. The Collector adds it.
• You want to filter or rename metrics before they reach SigNoz.
• You want to send the data to more than one backend.
• The broker cannot connect to your SigNoz instance directly.

Point EMQX at the gRPC port of the Collector, where <collector-host> is the host that runs it. Remove the TLS and header settings, because the Collector now holds the ingestion key:

opentelemetry {
  exporter { endpoint = "http://<collector-host>:4317" }
  metrics { enable = true, interval = "10s" }
  traces { enable = true, filter { trace_all = true } }
  logs { enable = true, level = warning }
}

Then add an OTLP receiver to your existing Collector configuration. Reference it from your metrics, traces, and logs pipelines:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317