Installing kafka and zookeeper cluster using kubernetes

5/9/2019

Can anyone share me the yaml file for creating kafka cluster with two kafka broker and zookeeper cluster with 3 servers.I'm new to kubernetes.

-- Radha
apache-kafka
kubernetes

4 Answers

5/10/2019

I think that you could take a look at the Strimzi project here https://strimzi.io/. It's based on the Kubernetes operator pattern and provide a simple way to deploy and manage a Kafka cluster on Kubernetes using custom resources. The Kafka cluster is described through a new "Kafka" resource YAML file for setting all you need. The operator takes care of that and deploys the Zookeeper ensemble + the Kafka cluster for you. It also deploys more two operators for handling topics and users (but they are optional).

-- ppatierno
Source: StackOverflow

11/22/2019

Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:

https://github.com/StanislavKo/k8s_digitalocean_kafka

You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (username/password).

Kafka cluster is StatefulSet, so you can scale cluster easily.

-- StanislavKo
Source: StackOverflow

5/10/2019

Take look at https://github.com/Yolean/kubernetes-kafka, Make sure the broker memory limit is 2 GB or above.

Maintaining a reliable kafka cluster in kubernetes is still a challenge, good luck.

-- silverfox
Source: StackOverflow

5/10/2019

I recommend you to try Strimzi Kafka Operator. Using it you can define a Kafka cluster just like other Kubernetes object - writing a yaml file. Moreover, also users, topics and Kafka Connect cluster are just a k8s objects. Some (by not all!) features of Strimzi Kafka Operator:

  • Secure communication between brokers and between brokers and zookeeper with TLS
  • Ability to expose the cluster outside k8s cluster
  • Deployable as a helm chart (it simplifies things a lot)
  • Rolling updates when changing cluster configuration
  • Smooth scaling out
  • Ready to monitor the cluster using Prometheus and Grafana.

It's worth to mention a great documentation.

Creating a Kafka cluster is as simple as applying a Kubernetes manifest like this:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.2.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.2"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
-- gkocur
Source: StackOverflow