kafka consumer issue: connection drop after consumer is created: Kafka + kubernetes

12/9/2019

I am trying to build the exact infra mentioned here

Basically, the article is specifically for scaling pods based on the reading the Stackdriver metrics(stack drive receives traffic from the Kafka server and scaling happens based on it.)

Everything is working fine until I create a producer deployment and it works fine as expected. It is able to connect to the Kafka server but when I run the consumer deployment, it is failing with the following error in the container logs.

% Error: 10.128.0.9:9092/bootstrap: Connect to ipv4#10.128.0.9:9092 failed: Operation timed out (after 128991ms in state CONNECT)

screenshot:
enter image description here I just have one GKE cluster with 2 nodes + one Kafka compute engine instance. There is no special setup. I am just following the above article.

I don't know what should I debug because the producer is working but the consumer is not.

If anyone can help that would be really appreciated.

-- Mitesh Gangaramani
apache-kafka
kubernetes
stackdriver

1 Answer

12/10/2019

it turned out a firewall issue. I was just allowing the node network whereas it required allowing the pod network and service network to function.

-- Mitesh Gangaramani
Source: StackOverflow