I am able to produce/consume message without any error from inside kubernetes But, I am not able to access from outside of kubernetes
Here is my setup:
kubectl get svc -o wide
kafka-test NodePort 10.108.201.154 <none> 9092:30989/TCP 1d app=kafka,release=kafka-test
kafka-test-0-external NodePort 10.97.62.11 <none> 9092:31090/TCP 1d app=kafka,release=kafka-test,statefulset.kubernetes.io/pod-name=kafka-test-0
kafka-test-1-external NodePort 10.99.211.234 <none> 9092:31091/TCP 1d app=kafka,release=kafka-test,statefulset.kubernetes.io/pod-name=kafka-test-1
kafka-test-2-external NodePort 10.105.160.163 <none> 9092:31092/TCP 1d app=kafka,release=kafka-test,statefulset.kubernetes.io/pod-name=kafka-test-2
kafka-test-headless ClusterIP None <none> 9092/TCP 1d app=kafka,release=kafka-test
kafka-test-zookeeper NodePort 10.106.113.172 <none> 2181:31541/TCP 1d app=zookeeper,release=kafka-test
kubectl get pods -o wide
kafka-test-0 1/1 Running 0 1d 192.168.2.229 ip-xx-xx-xx-xx.ap-south-1.compute.internal <none>
kafka-test-1 1/1 Running 0 1d 192.168.59.198 ip-xx-xx-xx-xx.ap-south-1.compute.internal <none>
kafka-test-2 1/1 Running 0 1d 192.168.4.9 ip-xx-xx-xx-xx.ap-south-1.compute.internal <none>
kafka-test-zookeeper-0 1/1 Running 0 1d 192.168.2.228 ip-xx-xx-xx-xx.ap-south-1.compute.internal <none>
kafka-test-zookeeper-1 1/1 Running 0 1d 192.168.59.199 ip-xx-xx-xx-xx.ap-south-1.compute.internal <none>
kafka-test-zookeeper-2 1/1 Running 0 1d 192.168.4.10 ip-xx-xx-xx-xx.ap-south-1.compute.internal <none>
I am running the 3 kafka service with following properties
/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties \
--override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT \
--override advertised.listeners=INTERNAL_PLAINTEXT://kafka-test-0.kafka-test-headless.spr-ops.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://<publicdns name of pod instance>:31090 \
--override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:31090 \
--override inter.broker.listener.name=INTERNAL_PLAINTEXT
/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties \
--override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT \
--override advertised.listeners=INTERNAL_PLAINTEXT://kafka-test-1.kafka-test-headless.spr-ops.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://<publicdns name of pod instance>:31091 \
--override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:31091 \
--override inter.broker.listener.name=INTERNAL_PLAINTEXT
/etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/server.properties \
--override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT \
--override advertised.listeners=INTERNAL_PLAINTEXT://kafka-test-2.kafka-test-headless.spr-ops.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://<publicdns name of pod instance>:31092 \
--override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:31092 \
--override inter.broker.listener.name=INTERNAL_PLAINTEXT
I am able to telnet to the instances using ports 31090,31091,31092 which are opened for external access.
I am able to list, describe topic from outside kubernetes, but produce/consume are not working from outside
when I tried to produce from outside k8s, I am getting below issue
./kafka-console-producer.sh --broker-list <publicdns name of pod instance>:31090,<publicdns name of pod instance>:31091,<publicdns name of pod instance>:31092 --topic spr
hi
[2019-01-14 11:38:50,623] ERROR Error when sending message to topic spr with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for spr-1: 1524 ms has passed since batch creation plus linger time
test
[2019-01-14 11:40:24,385] ERROR Error when sending message to topic spr with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for spr-0: 1518 ms has passed since batch creation plus linger time
I am stuck with this and trying from long time, any help here most appreciated.