Kafka and zookeper setup on kubernetes

1/20/2018

We are planning to set up Kafka and zookeeper on kubernetes, Zookeeper is working fine as expected, but while setting up Kafka, kafka is giving the issues, Kafka is not able to communicate zookepeer. We are planning to setup internal load balancer to communicate to external world, how can we communicate. Please refer to below files:

Kafka:

---
apiVersion: v1
kind: Service
metadata:
  name: kafka
  namespace: default
spec:
  ports:
  - port: 9092
  clusterIP:
  selector:
    app: kafka
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: broker
  namespace: default
spec:
  serviceName: "kafka"
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:3.2.0@sha256:82cb7a49161705ea750f3466700cce7a57e45d772f4bdbcddcdaac336dac77c0
        env:
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zookeeper:2181"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: PLAINTEXT://broker-0.kafka.cp-quickstart.svc.cluster.local:29092
        ports:
        - containerPort: 9092

Zookeeper cluster:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-31
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper-31
      name: zookeeper-31
  template:
    metadata:
      labels:
        app: zookeeper-31
        name: zookeeper-31
    spec:
      containers:
      - name: zoo31
        image: confluentinc/cp-zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "3"
        - name: ZOOKEEPER_SERVER_11
          value: zoo11
        - name: ZOOKEEPER_SERVER_21
          value: zoo21
        - name: ZOOKEEPER_SERVER_31
          value: zoo31
        - name: ZOOKEEPER_INIT_LIMIT
          value: "5"
        - name: ZOOKEEPER_SYNC_LIMIT
          value: "2"
        - name: ZOOKEEPER_SNAP_COUNT
          value: "100000"
        - name: ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT
          value: "10"
        - name: ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL
          value: "24"
        - name: ZOOKEEPER_CLIENT_PORT
          value: "2181"  

Zookeeper services:

apiVersion: v1
kind: Service
metadata:
  name: zoo12
  labels:
    app: zookeeper-11
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-11
---
apiVersion: v1
kind: Service
metadata:
  name: zoo21
  labels:
    app: zookeeper-21
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-21
---
apiVersion: v1
kind: Service
metadata:
  name: zoo31
  labels:
    app: zookeeper-31
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-31
-- Raju
kubernetes

1 Answer

1/21/2018
- name: KAFKA_ZOOKEEPER_CONNECT

must list all zookeeper servers with syntax:

statefullName.service

for example: zookeeper-0.zookeeper:2181

You don't need to create 3 services for zookeeper, one for each stateful. One is enough.

-- Nicola Ben
Source: StackOverflow