I have a landoop kafka image running on a Pod on minikube k8 cluster on my mac. I have 2 different services to expose the port 8081 for schema registry and 9092 for broker. I have mapped the ports 8081 -> 30081 and 9092 -> 30092 in my NodePort services so that I can access it from outside the cluster. But when I try to run a console consumer or my consumer app, Kafka never consumes messages. To verify broker 9092 port is reachable outside k8 cluster:
nc <exposed-ip> 30092, it says the port is open.To verify Schema registry 8081 is reachable:
curl -X GET http://192.168.99.100:30081/subjectsIt returns the schemas that are available.
I had a couple of questions. 1) Can we not access Kafka out of k8 cluster in an above-mentioned way outside of k8 cluster?If so am I doing it wrong in some way? 2) If the port is open, doesn't that mean that broker is available?
Any help is appreciated.Thanks
You need to set the advertised listeners for Kafka. For the landoop docker images this can be set via the environment flag
-e ADV_HOST=192.168.99.100
Accessing a Kafka cluster from outside a container network is rather complicated if you cannot route directly from the outside to the pod.
When you first connect to a Kafka cluster you connect to a single broker and the broker returns the list of all brokers and partitions inside the Kafka cluster. The Kafka client then uses the list to interact with the brokers where the specific topic lays.
The problem is that the broker lists contains by default the internal IP of the Kafka broker. Which would be in your case the container network ip. You can overwrite this value by setting advertised.listeners inside each broker's configuration.
To make a Kafka cluster available from outside Kubernetes you need to configure a nodeport service per each of your brokers and set the advertised.listeners setting of each broker to the external ip of the corresponding nodeport service. But note that this adds additional latency and failure points when you try to use Kafka from inside your Kubernetes cluster.