I'd like to setup 3-nodes zookeeper cluster with version 3.5.1. In order to deploy it into kubernetes, I have to make zookeeper node itself to be 0.0.0.0, so it can be started correctly. the configuration as below:
zk1: zoo.cfg
standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
zoo.cfg.dynamic
server.1=0.0.0.0:2888:3888:participant;2181
server.2=10.62.56.192:2889:3889:participant;2182
server.3=10.62.56.192:2890:3890:participant;2183
zk2: zoo.cfg
standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
zoo.cfg.dynamic
server.1=10.62.56.192:2888:3888:participant;2181
server.2=0.0.0.0:2889:3889:participant;2182
server.3=10.62.56.192:2890:3890:participant;2183
zk3: zoo.cfg
standaloneEnabled=false
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
zoo.cfg.dynamic
server.1=10.62.56.192:2888:3888:participant;2181
server.2=10.62.56.192:2889:3889:participant;2182
server.3=0.0.0.0:2890:3890:participant;2183
The result is only zk1 can be started, the zk2 and zk3 fails to start. The errors are
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: client address for this server (id = 2) in static config file is 0.0.0.0/0.0.0.0:2181 is different from client address found in dynamic file: /0.0.0.0:2182
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupClientPort(QuorumPeerConfig.java:608)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:506)
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:157)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:110)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Invalid config, exiting abnormally
The point is that we have to configure the node itself as 0.0.0.0.
Try: https://hub.docker.com/r/elevy/zookeeper/ which is based on https://github.com/eliaslevy/docker-zookeeper
That is using 1 service per zk pod. Each service has its own "hostname", which can be resolved via DNS from within each zk pod. The names are zookeeper-1, zookeeper-2, and zookeeper-3.
So, instead of config lines like
server.1=0.0.0.0:2888:3888:participant;2181
server.2=10.62.56.192:2889:3889:participant;2182
server.3=10.62.56.192:2890:3890:participant;2183
you have config lines like:
server.1=zookeeper-1:2888:3888
server.2=zookeeper-2:2888:3888
server.3=zookeeper-3:2888:3888
Note that since each Pod has its own IP, you don't need to use a range of ports, like 2888, 2889, 2890. You can just use 2888/3888 for each one.