Create a separate cluster for running SigNoz. This will help in the isolation of application and APM environments and hence, reduce the impact radius of operational issues.
Use infra-level otel-collectors to send host metrics from VMs (should be part of default setup)
Configure TTL for disk and use move to s3 for reduced costs. Perf of s3 is 2-3x slower than EBS. Configure retention for each of metrics, traces and logs.
Setup alerts on important APM metrics
Harness the power of distributed tracing data by creating dashboards using Clickhouse queries. You can run group by and aggregates on tags(attributeMap) and events of a span. Also, filtering by more specific conditions should be possible. Let us know if you would like us to help write a few queries to plot a chart using the traces data. Same also, applies for the logs data.
Secure query-service and otel-collector using TLS ingress
Authorise client otel-collectors to send data to signoz cluster (planned)
otel-collectorwhich works on the push model and not
otel-collector-metricswhich works on the pull model of prometheus scraping. You need to add a different config to add another instance of
otel-collector-metricsto prevent duplication
Use higher batch size in
otel-collectorwhen ingesting more than 10K events/s. The default batch size is 10K rows. Batch size upto 50K should work well.
Use sampling to reduce the amount of data sent to SigNoz