I am working on deploying Kafka/Zookeeper in Kubernetes using MINIKUBE. below is my YAML file:
##################################
# Setup Zookeeper Deployment
##################################
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: zookeeper
name: zookeeper
spec:
replicas: 1
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- image: wurstmeister/zookeeper
# imagePullPolicy: Always
name: zookeeper
ports:
- containerPort: 2181
##################################
# Setup Zookeeper Service
##################################
---
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
spec:
type: NodePort
ports:
- name: zookeeper-port
port: 2181
nodePort: 30181
targetPort: 2181
selector:
app: zookeeper
---
##################################
# Setup Kafka service
##################################
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
ports:
- name: kafka-port
port: 9092
nodePort: 30092
targetPort: 9092
selector:
app: kafka
---
##################################
# Setup Kafka Broker Deployment
##################################
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: kafka
name: kafka
spec:
replicas: 1
template:
metadata:
labels:
app: kafka
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: 192.168.99.100
- name: KAFKA_ADVERTISED_PORT
value: "30092"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: 192.168.99.100:30181
- name: KAFKA_ADVERTISED_LISTENERS
value: "PLAINTEXT://192.168.99.100:30092"
# - name: KAFKA_LISTENERS
# value: "PLAINTEXT://192.168.99.100:9092"
- name: KAFKA_CREATE_TOPICS
value: "vignesh-topic:1:1"
- name: LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER
value: "DEBUG"
image: wurstmeister/kafka
#imagePullPolicy: Always
name: kafka
ports:
- containerPort: 9092
I have successfully created the Deployment/Services in local machine Kubernetes using MINIKUBE using below command.
kubectl create -f kafka.yml
I have navigated inside Kafka pods and I am able to create a topic using below command,
./bin/kafka-topics.sh --create --zookeeper 192.168.99.100:30181 --replication-factor 1 --partitions 1 --topic test-topic
But, When I try to send a message to the topic (test-topic), the system throws the below error.
Note
when I run netstat -tunap , both port 30092 and 30181 is showing established.
I don't know what I am missing here. Please help me to move forward.
Thanks and Appreciate your help.
Thank you @SoheilPourbafrani and @cricket_007 for your help! I have found the workaround for the question I asked above.
Once I run the below command in the window PowerShell, Kafka started properly and able to communicate with it from Node Application and Kafka Tool as well.
minikube ssh
sudo ip link set docker0 promisc on
References: Newer versions of Minikube don't allow Pods to use their own Services