Troubleshooting

This troubleshooting guide includes step-by-step instructions that should resolve most installation issues.

Using SigNoz Troubleshooting Repository

You can use the SigNoz troubleshoot to test if SigNoz otel collector is accessible from where you are running your applications.

Binary installation

You can run the one liner script below to download the troubleshoot binary:

curl -sL https://github.com/SigNoz/troubleshoot/raw/main/scripts/install.sh | bash

Here is the syntax:

./troubleshoot checkEndpoint --endpoint=<endpoint-to-check>

Binary

For example, if Otel Collector should be accessible in localhost:4317:

./troubleshoot checkEndpoint --endpoint=localhost:4317

Docker

You can also use troubleshoot Docker image:

docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=172.17.0.1:4317

Kubernetes

Let's spin up a pod in Kubernetes with platform namespace to check if otel collector is running properly:

kubectl -n platform run troubleshoot --image=signoz/troubleshoot \
  --restart='Never' -i --tty --rm --command -- ./troubleshoot checkEndpoint \
  --endpoint=my-release-signoz-otel-collector.platform.svc.cluster.local:4317

You can also spin up a pod in Kubernetes with same namespace as your application to check if otel collector is accessible:

kubectl -n app-namespace run troubleshoot --image=signoz/troubleshoot \
  --restart='Never' -i --tty --rm --command -- ./troubleshoot checkEndpoint \
  --endpoint=my-release-signoz-otel-collector.platform.svc.cluster.local:4317

helm upgrade --install -n platform my-release signoz/signoz \
  --set otelCollector.serviceType="<NodePort or LoadBalancer>"

Troubleshooting Video

If you instead prefer to watch a video, check out this video:

Troubleshooting Docker Standalone Installation of SigNoz

1. Before you install SigNoz on Docker Standalone, ensure that all prerequisites are met.
2. Run the docker ps command and ensure that the signoz-clickhouse and signoz-query-service containers are running. If these containers are not running, increase the memory allocated to Docker.
3. If you don't see any data in the front-end, please wait two or three minutes.
4. Run the install.sh script again. The script will try to reinstall the containers that failed.
5. Follow the steps for uninstalling SigNoz section and then install SigNoz again by following the steps in the Install SigNoz on Docker Standalone section.
6. If you're still facing issues trying to install SigNoz, please reach out to us on Slack

Troubleshooting ClickHouse container crashes

If the clickhouse container is unable to start due to certain reasons, it is recommended to take the following steps before recreating the container

1. Run the following command to see more detailed logs for the ClickHouse container:

sudo docker-compose -f ./docker/clickhouse-setup/docker-compose.yaml ps -a

This will show you the status of all containers, including ClickHouse.

• Verify system resources: Ensure your system has enough resources (CPU, memory, disk space) to run the ClickHouse container.

• Check permissions: Make sure you have the necessary permissions to run Docker containers. Try running the installation script with sudo privileges.

• Review Docker configuration: Ensure your Docker installation is up-to-date and properly configured.

2. If the logs mention about Permission denied error, it is likely due to the correct permissions not set for the container.
3. Use the chwon command to change the ownership for the files that clickhouse container is unable to access.
4. If neither of this works, run the install.sh script again as the root user or by using sudo as a prefix.

SigNoz Otel Collector Address Grid

You might have specific set up for your application and SigNoz cluster. It might not be very clear on which address to use to send data to SigNoz.

Here is the SigNoz Otel Collector address grid which could be helpful:

helm upgrade --install -n platform my-release signoz/signoz \
--set otelCollector.serviceType="<NodePort or LoadBalancer>"