Best Practices to follow to run SigNoz in production

1. Create a separate cluster for running SigNoz. This will help in the isolation of application and APM environments and hence, reduce the impact radius of operational issues.

2. If you are running in k8s, use k8s-infra to collect telemetry (logs, metrics, traces).

3. Configure TTL for disk and use move to s3 for reduced costs. Perf of s3 is 2-3x slower than EBS. Configure retention for each of metrics, traces and logs. See Retention Period | SigNoz

4. Setup alerts on important APM metrics

5. Harness the power of distributed tracing data by creating dashboards using Clickhouse queries. You can run group by and aggregates on tags(attributeMap) and events of a span. Also, filtering by more specific conditions should be possible. Let us know if you would like us to help write a few queries to plot a chart using the traces data. Same also, applies for the logs data.

6. Secure signoz and otel-collector using TLS ingress. See Secure SigNoz in Kubernetes using Ingress-NGINX and Cert-Manager | SigNoz

7. Horizontally scale otel-collector which works on the push model.

8. Use higher batch size in otel-collector when ingesting more than 10K events/s. The default batch size is 10K rows. Batch size upto 50K should work well.

9. Use sampling to reduce the amount of data sent to SigNoz. See opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor