Distributed ClickHouse with Docker Swarm

To set up ClickHouse cluster with 3 shards in Docker Swarm, you will need to include additional clickhouse services in docker-compose.yaml. You will also need to update clickhouse-cluster.xml to include those additional clickhouse services as multiple shards.

That can be done in the by un-commenting the following in docker-compose.yaml:

x-clickhouse-depend: &clickhouse-depend
  depends_on:
    clickhouse:
      condition: service_healthy
    clickhouse-2:
      condition: service_healthy
    clickhouse-3:
      condition: service_healthy

services:
  ...

  clickhouse-2:
    <<: *clickhouse-defaults
    hostname: clickhouse-2
    ports:
      - "9001:9000"
      - "8124:8123"
      - "9182:9181"
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
      - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
      - ./data/clickhouse-2/:/var/lib/clickhouse/

  clickhouse-3:
    <<: *clickhouse-defaults
    hostname: clickhouse-3
    ports:
      - "9002:9000"
      - "8125:8123"
      - "9183:9181"
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
      - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
      - ./data/clickhouse-3/:/var/lib/clickhouse/

To switch to 3 nodes Zookeepers cluster from default 1 node Zookeeper, un-comment the following in docker-compose.yaml:

x-clickhouse-defaults: &clickhouse-defaults
  ...
  depends_on:
    - zookeeper-1
    - zookeeper-2
    - zookeeper-3

services:
  zookeeper-1:
    ...
    environment:
      - ZOO_SERVER_ID=1
      - ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper-2:2888:3888,zookeeper-3:2888:3888
      ...

  zookeeper-2:
    image: bitnami/zookeeper:3.7.0
    hostname: zookeeper-2
    user: root
    ports:
      - "2182:2181"
      - "2889:2888"
      - "3889:3888"
    volumes:
      - ./data/zookeeper-2:/bitnami/zookeeper
    environment:
      - ZOO_SERVER_ID=2
      - ZOO_SERVERS=zookeeper-1:2888:3888,0.0.0.0:2888:3888,zookeeper-3:2888:3888
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_AUTOPURGE_INTERVAL=1

  zookeeper-3:
    image: bitnami/zookeeper:3.7.0
    hostname: zookeeper-3
    user: root
    ports:
      - "2183:2181"
      - "2890:2888"
      - "3890:3888"
    volumes:
      - ./data/zookeeper-3:/bitnami/zookeeper
    environment:
      - ZOO_SERVER_ID=3
      - ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_AUTOPURGE_INTERVAL=1

Next, you will have to un-comment the following from clickhouse-cluster.xml:

<clickhouse>
    <zookeeper>
        ...
        <node index="2">
            <host>zookeeper-2</host>
            <port>2181</port>
        </node>
        <node index="3">
            <host>zookeeper-3</host>
            <port>2181</port>
        </node>
    </zookeeper>

    <remote_servers>
        <cluster>
            ...
            <shard>
                <replica>
                    <host>clickhouse-2</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>clickhouse-3</host>
                    <port>9000</port>
                </replica>
            </shard>
        </cluster>
    </remote_servers>
</clickhouse>

Next, you will have to toggle DOCKER_MULTI_NODE_CLUSTER environment variable to true to ensure migrations are run on new instances (shards) of clickhouse.

services:  
  otel-collector:
    environment:
      - DOCKER_MULTI_NODE_CLUSTER=true

Finally, we execute the following to apply updated docker-compose.yaml:

cd deploy/docker-swarm

docker stack deploy -c docker-compose.yaml signoz

(Optional) After the migration files run once in all clickhouse instances and healthy SigNoz cluster is verified, you need to make sure migration files do not run for every otel-collector container restart.

You can do that by toggling back DOCKER_MULTI_NODE_CLUSTER environment variable back to false.

services:  
  otel-collector:
    environment:
      - DOCKER_MULTI_NODE_CLUSTER=false

Followed by executing the commands below to apply updated docker-compose.yaml:

cd deploy/docker-swarm

docker stack deploy -c docker-compose.yaml signoz

Was this page helpful?

Docker

Kubernetes