Distributed ClickHouse with Docker
For ClickHouse cluster with 3 shards, you will need to include additional clickhouse services in docker-compose.yaml. You will also need to update clickhouse-cluster.xml to include those additional clickhouse services as multiple shards.
That can be done in the by un-commenting the following in docker-compose.yaml:
x-clickhouse-depend: &clickhouse-depend
depends_on:
clickhouse:
condition: service_healthy
clickhouse-2:
condition: service_healthy
clickhouse-3:
condition: service_healthy
services:
...
clickhouse-2:
<<: *clickhouse-defaults
container_name: signoz-clickhouse-2
hostname: clickhouse-2
ports:
- "9001:9000"
- "8124:8123"
- "9182:9181"
volumes:
- ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
- ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
- ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
# - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
- ./data/clickhouse-2/:/var/lib/clickhouse/
clickhouse-3:
<<: *clickhouse-defaults
container_name: signoz-clickhouse-3
hostname: clickhouse-3
ports:
- "9002:9000"
- "8125:8123"
- "9183:9181"
volumes:
- ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
- ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
- ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
# - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
- ./data/clickhouse-3/:/var/lib/clickhouse/
To switch to 3 nodes Zookeepers cluster from default 1 node Zookeeper, un-comment the following in docker-compose.yaml:
x-clickhouse-defaults: &clickhouse-defaults
...
depends_on:
- zookeeper-1
- zookeeper-2
- zookeeper-3
services:
zookeeper-1:
...
environment:
- ZOO_SERVER_ID=1
- ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper-2:2888:3888,zookeeper-3:2888:3888
...
zookeeper-2:
image: bitnami/zookeeper:3.7.0
container_name: signoz-zookeeper-2
hostname: zookeeper-2
user: root
ports:
- "2182:2181"
- "2889:2888"
- "3889:3888"
volumes:
- ./data/zookeeper-2:/bitnami/zookeeper
environment:
- ZOO_SERVER_ID=2
- ZOO_SERVERS=zookeeper-1:2888:3888,0.0.0.0:2888:3888,zookeeper-3:2888:3888
- ALLOW_ANONYMOUS_LOGIN=yes
- ZOO_AUTOPURGE_INTERVAL=1
zookeeper-3:
image: bitnami/zookeeper:3.7.0
container_name: signoz-zookeeper-3
hostname: zookeeper-3
user: root
ports:
- "2183:2181"
- "2890:2888"
- "3890:3888"
volumes:
- ./data/zookeeper-3:/bitnami/zookeeper
environment:
- ZOO_SERVER_ID=3
- ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
- ALLOW_ANONYMOUS_LOGIN=yes
- ZOO_AUTOPURGE_INTERVAL=1
Next, you will have to un-comment the following from clickhouse-cluster.xml:
<clickhouse>
<zookeeper>
...
<node index="2">
<host>zookeeper-2</host>
<port>2181</port>
</node>
<node index="3">
<host>zookeeper-3</host>
<port>2181</port>
</node>
</zookeeper>
<remote_servers>
<cluster>
...
<shard>
<replica>
<host>clickhouse-2</host>
<port>9000</port>
</replica>
</shard>
<shard>
<replica>
<host>clickhouse-3</host>
<port>9000</port>
</replica>
</shard>
</cluster>
</remote_servers>
</clickhouse>
Next, you will have to toggle DOCKER_MULTI_NODE_CLUSTER environment variable to true to ensure migrations are run on new instances (shards) of clickhouse.
services:
otel-collector:
environment:
- DOCKER_MULTI_NODE_CLUSTER=true
Finally, we execute the following to apply updated docker-compose.yaml:
cd deploy
./install.sh
(Optional) After the migration files run once in all clickhouse instances and healthy SigNoz cluster is verified, you need to make sure migration files do not run for every otel-collector container restart.
You can do that by toggling back DOCKER_MULTI_NODE_CLUSTER environment variable back to false.
services:
otel-collector:
environment:
- DOCKER_MULTI_NODE_CLUSTER=false
Followed by executing the commands below to apply updated docker-compose.yaml:
cd deploy
./install.sh
