How to access kafka installed outside kubernetes cluster from a service provisioned inside a kubernetes cluster

5/18/2020

My setup is like, I have a producer service provisioned as part of minikube cluster and it is trying to publish messages to a kafka instance running on the host machine.

I have written a kafka service and endpoints yaml as follows:

kind: Service
apiVersion: v1
metadata:
  name: kafka
spec:
  ports:
    - name: "broker"
      protocol: "TCP"
      port: 9092
      targetPort:  9092
      nodePort: 0
---
kind: Endpoints
apiVersion: v1
metadata:
  name: kafka
  namespace: default
subsets:
  - addresses:
      - ip: 10.0.2.2
    ports:
      - name: "broker"
        port: 9092

The ip address of the host machine from inside the minikube cluster mentioned in the endpoint is acquired from the following command:

minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print \$2 }'"

The problem I am facing is that the topic is getting created when producer tries to publish message for the first time but no messages are getting written on to that topic.

Digging into the pod logs, I found that producer is trying to connect to kafka instance on localhost or something (not really sure of it).

2020-05-17T19:09:43.021Z [warn] org.apache.kafka.clients.NetworkClient [] - 
[Producer clientId=/system/sharding/kafkaProducer-greetings/singleton/singleton/producer] 
Connection to node 0 (omkara/127.0.1.1:9092) could not be established. Broker may not be available.

Following which I suspected that probably I need to modify server.properties with the following change:

listeners=PLAINTEXT://localhost:9092

This however resulted in the change in the ip address in the log:

2020-05-17T19:09:43.021Z [warn] org.apache.kafka.clients.NetworkClient [] - 
[Producer clientId=/system/sharding/kafkaProducer-greetings/singleton/singleton/producer] 
Connection to node 0 (omkara/127.0.0.1:9092) could not be established. Broker may not be available.

I am not sure what ip address must be mentioned here? Or what is an alternate solution? And if it is even possible to connect from inside the kubernetes cluster to the kafka instance installed outside the kubernetes cluster.

-- Shivam Kapoor
apache-kafka
kubernetes

1 Answer

5/20/2020

Since producer kafka client is on the very same network as the brokers, we need to configure an additional listener like so:

listeners=INTERNAL://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://localhost:9093,EXTERNAL://10.0.2.2:9092
inter.broker.listener.name=INTERNAL

We can verify messages in topic like so:

kafka-console-consumer.sh --bootstrap-server INTERNAL://0.0.0.0:9093 --topic greetings --from-beginning
{"name":"Alice","message":"Namastey"}

You can find a detailed explaination on understanding & provisioning kafka listeners here.

-- Shivam Kapoor
Source: StackOverflow