Can anyone share me the yaml file for creating kafka cluster with two kafka broker and zookeeper cluster with 3 servers.I'm new to kubernetes.
I think that you could take a look at the Strimzi project here https://strimzi.io/. It's based on the Kubernetes operator pattern and provide a simple way to deploy and manage a Kafka cluster on Kubernetes using custom resources. The Kafka cluster is described through a new "Kafka" resource YAML file for setting all you need. The operator takes care of that and deploys the Zookeeper ensemble + the Kafka cluster for you. It also deploys more two operators for handling topics and users (but they are optional).
Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:
https://github.com/StanislavKo/k8s_digitalocean_kafka
You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (username/password).
Kafka cluster is StatefulSet, so you can scale cluster easily.
Take look at https://github.com/Yolean/kubernetes-kafka, Make sure the broker memory limit is 2 GB or above.
Maintaining a reliable kafka cluster in kubernetes is still a challenge, good luck.
I recommend you to try Strimzi Kafka Operator. Using it you can define a Kafka cluster just like other Kubernetes object - writing a yaml file. Moreover, also users, topics and Kafka Connect cluster are just a k8s objects. Some (by not all!) features of Strimzi Kafka Operator:
It's worth to mention a great documentation.
Creating a Kafka cluster is as simple as applying a Kubernetes manifest like this:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.2.0
replicas: 3
listeners:
plain: {}
tls: {}
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.2"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}