K
Q

Errors while deploying Kafka on Kubernetes

8/8/2018

I have a Kafka setup on Kubernetes with two brokers running on two services.

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka-service    NodePort    10.101.47.36     <none>        9092:32132/TCP               2h
kafka-service2   NodePort    10.110.151.216   <none>        9092:32133/TCP               2h
kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP                      1d
zoo1             ClusterIP   10.105.248.153   <none>        2181/TCP,2888/TCP,3888/TCP   2h

Both of these are nodeports exposed on 32132 and 32133 ports respectively.

I created the cluster with the following spec:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker1
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "1"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "32133"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: localhost
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "1"
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker2
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "1"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "32132"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: localhost
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181

This spinned up two pods kafka-broker1 and kafka-broker2. I have a separate service for Zookeeper running called zoo1.

I tried to create a new topic with the following command:

bin/kafka-topics.sh --create --zookeeper 10.105.248.153:2181 --replication-factor 2 --partitions 2 --topic test

And it got created. Below is the description of topic test

Topic:test  PartitionCount:2    ReplicationFactor:2 Configs:
    Topic: test Partition: 0    Leader: 2   Replicas: 2,1   Isr: 2
    Topic: test Partition: 1    Leader: 2   Replicas: 1,2   Isr: 2

But when I try to publish messages to this topic through a producer:

kafka/bin/kafka-console-producer.sh --broker-list 10.110.151.216:9092,10.101.47.36:9092 --topic test OR kafka/bin/kafka-console-producer.sh --broker-list localhost:32132,localhost:32133 --topic test

I keep getting the following warnings:

[2018-08-08 09:53:31,629] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:31,731] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:31,840] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:31,949] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,056] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,163] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 7 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,272] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 8 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,382] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 9 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,488] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 10 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

I have no idea why this is happening.

-- Piyush Shrivastava
apache-kafka
kubernetes

Similar Questions

Failed to discover supported resources
How to mount a single file in container in kubernetes/docker
istio egressgateway: routing through a static IP
How to connect to a third party VPN from Google Cloud Platform?
Spinnaker is not able to download helm charts from Artifactory

2 Answers

10/31/2019

Ensure that Kafka external port and k8s service nodePort are consistent, Other services call k8s-service:nodeport. write this config_kafka_in_kubernetes. i answer this question kafka-behind-traefik , hope to help U !

-- bbotte
Source: StackOverflow
8/9/2018

Kafka uses Zookeeper to do leadership election of Kafka Broker and Topic Partition pairs. ZooKeeper returns the address of POD. In case of POD is recreated, ZooKeeper will return incorrect address. So the best practice is not yo use IP addresses of POD.

You’re using wurstmeister/kafka image, and I found in it’s Readme one note about KAFKA_ADVERTISED_HOST_NAME:

Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.

You have 2 brokers in your installation and each of them with localhost as a host name. I think that can be a reason of your problem. Try to use kafka1 and kafka2 instead of localhost and add aliases for that name to /etc/hosts file:

kafka1 127.0.0.1 kafka2 127.0.0.1

Then, you can try to connect to brokers like:

kafka/bin/kafka-console-producer.sh --broker-list kafka1:32132,kafka2:32133 --topic test

Also, looks like KAFKA_BROKER_ID settings are missing in the second deployment, but it could be a copy/paste issue.

-- Akar
Source: StackOverflow