We are planning to set up Kafka and zookeeper on kubernetes, Zookeeper is working fine as expected, but while setting up Kafka, kafka is giving the issues, Kafka is not able to communicate zookepeer. We are planning to setup internal load balancer to communicate to external world, how can we communicate. Please refer to below files:
Kafka:
---
apiVersion: v1
kind: Service
metadata:
name: kafka
namespace: default
spec:
ports:
- port: 9092
clusterIP:
selector:
app: kafka
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: broker
namespace: default
spec:
serviceName: "kafka"
replicas: 1
template:
metadata:
labels:
app: kafka
spec:
terminationGracePeriodSeconds: 10
containers:
- name: kafka
image: confluentinc/cp-kafka:3.2.0@sha256:82cb7a49161705ea750f3466700cce7a57e45d772f4bdbcddcdaac336dac77c0
env:
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://broker-0.kafka.cp-quickstart.svc.cluster.local:29092
ports:
- containerPort: 9092
Zookeeper cluster:
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: zookeeper-deployment-31
spec:
replicas: 1
selector:
matchLabels:
app: zookeeper-31
name: zookeeper-31
template:
metadata:
labels:
app: zookeeper-31
name: zookeeper-31
spec:
containers:
- name: zoo31
image: confluentinc/cp-zookeeper
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_ID
value: "3"
- name: ZOOKEEPER_SERVER_11
value: zoo11
- name: ZOOKEEPER_SERVER_21
value: zoo21
- name: ZOOKEEPER_SERVER_31
value: zoo31
- name: ZOOKEEPER_INIT_LIMIT
value: "5"
- name: ZOOKEEPER_SYNC_LIMIT
value: "2"
- name: ZOOKEEPER_SNAP_COUNT
value: "100000"
- name: ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT
value: "10"
- name: ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL
value: "24"
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"
Zookeeper services:
apiVersion: v1
kind: Service
metadata:
name: zoo12
labels:
app: zookeeper-11
spec:
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper-11
---
apiVersion: v1
kind: Service
metadata:
name: zoo21
labels:
app: zookeeper-21
spec:
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper-21
---
apiVersion: v1
kind: Service
metadata:
name: zoo31
labels:
app: zookeeper-31
spec:
ports:
- name: client
port: 2181
protocol: TCP
- name: follower
port: 2888
protocol: TCP
- name: leader
port: 3888
protocol: TCP
selector:
app: zookeeper-31
- name: KAFKA_ZOOKEEPER_CONNECT
must list all zookeeper servers with syntax:
statefullName.service
for example: zookeeper-0.zookeeper:2181
You don't need to create 3 services for zookeeper, one for each stateful. One is enough.