kafka - ERROR Error when sending message to topic test-topic with key: null, value: 17 bytes with error

10/24/2018

I am working on deploying Kafka/Zookeeper in Kubernetes using MINIKUBE. below is my YAML file:

##################################
#   Setup Zookeeper Deployment
##################################

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper
  name: zookeeper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - image: wurstmeister/zookeeper
       # imagePullPolicy: Always
        name: zookeeper
        ports:
        - containerPort: 2181


##################################
#   Setup Zookeeper Service
##################################

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-service
  name: zookeeper-service
spec:
  type: NodePort
  ports:
  - name: zookeeper-port
    port: 2181
    nodePort: 30181
    targetPort: 2181
  selector:
    app: zookeeper

---
##################################
  #   Setup Kafka service
##################################

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  ports:
  - name: kafka-port
    port: 9092
    nodePort: 30092
    targetPort: 9092
  selector:
    app: kafka

---
##################################
  #   Setup Kafka Broker Deployment
##################################

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kafka
  name: kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.99.100
        - name: KAFKA_ADVERTISED_PORT
          value: "30092"
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 192.168.99.100:30181    
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "PLAINTEXT://192.168.99.100:30092"
     #   - name: KAFKA_LISTENERS
     #     value: "PLAINTEXT://192.168.99.100:9092"
        - name: KAFKA_CREATE_TOPICS
          value: "vignesh-topic:1:1"
        - name: LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER
          value: "DEBUG"
        image: wurstmeister/kafka
        #imagePullPolicy: Always
        name: kafka
        ports:
        - containerPort: 9092

I have successfully created the Deployment/Services in local machine Kubernetes using MINIKUBE using below command.

kubectl create -f kafka.yml

I have navigated inside Kafka pods and I am able to create a topic using below command,

./bin/kafka-topics.sh --create --zookeeper 192.168.99.100:30181 --replication-factor 1 --partitions 1 --topic test-topic

But, When I try to send a message to the topic (test-topic), the system throws the below error.

error

Note

when I run netstat -tunap , both port 30092 and 30181 is showing established.

netstat

I don't know what I am missing here. Please help me to move forward.

Thanks and Appreciate your help.

-- Vignesh
apache-kafka
apache-zookeeper
kafka-topic
kubernetes
minikube

1 Answer

12/5/2018

Thank you @SoheilPourbafrani and @cricket_007 for your help! I have found the workaround for the question I asked above.

Once I run the below command in the window PowerShell, Kafka started properly and able to communicate with it from Node Application and Kafka Tool as well.

minikube ssh
sudo ip link set docker0 promisc on 

References: Newer versions of Minikube don't allow Pods to use their own Services

-- Vignesh
Source: StackOverflow