Production Readiness Checklist for Self-Hosted SigNoz

Overview

This checklist covers the operational work of running self-hosted SigNoz in production. Each section is independent, so you can apply them in any order.

Isolate the SigNoz deployment

Run SigNoz on a cluster separate from the applications it monitors. This separation limits the impact radius. An application incident cannot take down the tooling you need to debug that incident.

On Kubernetes, collect logs, metrics, and traces with k8s-infra.

Configure retention and storage

Set a retention period for each signal separately. Metrics, traces, and logs each have their own setting. See Retention Period.

To keep telemetry beyond your retention window for compliance or audit, add the awss3 exporter to fan newly ingested data out to S3. The S3 copy is not queryable from the SigNoz UI, so reduce your retention separately to lower storage cost. See Infinite Retention using AWS S3.

Secure SigNoz and the Collector

Put SigNoz and the OpenTelemetry Collector behind a TLS ingress. See Secure SigNoz in Kubernetes using Ingress-NGINX and Cert-Manager.

To enable TLS on individual Collector receivers and exporters, see Enable TLS for OpenTelemetry Collector.

Scale the OpenTelemetry Collector

The Collector works on a push model, so scale it horizontally. Add replicas as ingestion volume grows.

Above 10,000 events per second, raise the batch size. Update the batch processor in your existing otel-collector-config.yaml:

processors:
  batch:
    send_batch_size: 50000

Control ingestion volume

High ingestion volume increases ClickHouse storage and slows queries. Find which signal and service drives the volume first, then apply the matching fix.

• Cost Control: The full optimization flow, from analysis to validation.
• Optimize Metrics: Drop unused metrics and aggregate high-cardinality labels.
• Optimize Traces: Reduce trace volume with head-based or tail-based sampling.
• Optimize Logs: Filter verbose log records at the Collector.

Set up dashboards and alerts

Configure alerts on your key APM metrics: request rate, error rate, and latency.

Build dashboards from your trace and log data with ClickHouse queries. You can group by and aggregate on span attributes and span events, and filter on specific conditions. See ClickHouse queries for building dashboards and alerts.

Next Steps

• SigNoz Resources Planning: Size each SigNoz component for your expected volume.
• Upgrade Guides: Keep your deployment current.

Get Help

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.