Kafka on Kubernetes - not able to produce/consume outside of kubernetes

1/14/2019

I am able to produce/consume message without any error from inside kubernetes But, I am not able to access from outside of kubernetes

Here is my setup:

kubectl get svc -o wide

kafka-test                         NodePort    10.108.201.154   <none>        9092:30989/TCP                  1d    app=kafka,release=kafka-test
kafka-test-0-external              NodePort    10.97.62.11      <none>        9092:31090/TCP                  1d    app=kafka,release=kafka-test,statefulset.kubernetes.io/pod-name=kafka-test-0
kafka-test-1-external              NodePort    10.99.211.234    <none>        9092:31091/TCP                  1d    app=kafka,release=kafka-test,statefulset.kubernetes.io/pod-name=kafka-test-1
kafka-test-2-external              NodePort    10.105.160.163   <none>        9092:31092/TCP                  1d    app=kafka,release=kafka-test,statefulset.kubernetes.io/pod-name=kafka-test-2
kafka-test-headless                ClusterIP   None             <none>        9092/TCP                        1d    app=kafka,release=kafka-test
kafka-test-zookeeper               NodePort    10.106.113.172   <none>        2181:31541/TCP                  1d    app=zookeeper,release=kafka-test
kubectl get pods -o wide

kafka-test-0                                        1/1     Running            0          1d    192.168.2.229    ip-xx-xx-xx-xx.ap-south-1.compute.internal     <none>
kafka-test-1                                        1/1     Running            0          1d    192.168.59.198   ip-xx-xx-xx-xx.ap-south-1.compute.internal     <none>
kafka-test-2                                        1/1     Running            0          1d    192.168.4.9      ip-xx-xx-xx-xx.ap-south-1.compute.internal   <none>
kafka-test-zookeeper-0                              1/1     Running            0          1d    192.168.2.228    ip-xx-xx-xx-xx.ap-south-1.compute.internal     <none>
kafka-test-zookeeper-1                              1/1     Running            0          1d    192.168.59.199   ip-xx-xx-xx-xx.ap-south-1.compute.internal     <none>
kafka-test-zookeeper-2                              1/1     Running            0          1d    192.168.4.10     ip-xx-xx-xx-xx.ap-south-1.compute.internal   <none>

I am running the 3 kafka service with following properties

/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties \
   --override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT \
      --override advertised.listeners=INTERNAL_PLAINTEXT://kafka-test-0.kafka-test-headless.spr-ops.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://<publicdns name of pod instance>:31090 \
      --override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:31090 \
      --override inter.broker.listener.name=INTERNAL_PLAINTEXT
/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties \
   --override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT \
      --override advertised.listeners=INTERNAL_PLAINTEXT://kafka-test-1.kafka-test-headless.spr-ops.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://<publicdns name of pod instance>:31091 \
      --override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:31091 \
      --override inter.broker.listener.name=INTERNAL_PLAINTEXT
/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties \
   --override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT \
      --override advertised.listeners=INTERNAL_PLAINTEXT://kafka-test-2.kafka-test-headless.spr-ops.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://<publicdns name of pod instance>:31092 \
      --override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:31092 \
      --override inter.broker.listener.name=INTERNAL_PLAINTEXT

I am able to telnet to the instances using ports 31090,31091,31092 which are opened for external access.

I am able to list, describe topic from outside kubernetes, but produce/consume are not working from outside

when I tried to produce from outside k8s, I am getting below issue

./kafka-console-producer.sh --broker-list <publicdns name of pod instance>:31090,<publicdns name of pod instance>:31091,<publicdns name of pod instance>:31092 --topic spr

hi
[2019-01-14 11:38:50,623] ERROR Error when sending message to topic spr with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for spr-1: 1524 ms has passed since batch creation plus linger time
test
[2019-01-14 11:40:24,385] ERROR Error when sending message to topic spr with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for spr-0: 1518 ms has passed since batch creation plus linger time

I am stuck with this and trying from long time, any help here most appreciated.

-- IMRAN SHAIK
apache-kafka
kubernetes

0 Answers