We have a Kafka cluster in ec2. We have ec2 ip's as listeners and advertised.listeners in server.properties. (we don't have DNS resolution yet). We have MS in ec2 and k8s both with can connect to this kafka cluster and operate.

Now i am trying to setup istio in k8s. I have created DNS entry in route53 which will resolved to all kafka ip's and added ServiceEntry in istio. which should open port from envoy to kafka.

The problem is I can do kafka topic listing from istio but can not produce or consume.. DNS is getting resolved and kafka port is open from istio

I am suspecting we need to change advertised.listeners to DNS or i have put ip of all cluster. I couden't find any solution from istio to kafka. Also zookeeper service entry is not added do i have to add that also i am i am only connecting to kafka using sarama lib?

Any help is appreciated.

kakfa is 1.0.1 istio is 1.1.7

I have added producer error.

[2019-06-21 12:41:13,494] WARN [Producer clientId=console-producer] Got error produce response with correlation id 26 on topic-partition test-0, retrying (2 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2019-06-21 12:41:13,495] WARN [Producer clientId=console-producer] Received unknown topic or partition error in produce request on partition test-0. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender)
[2019-06-21 12:41:13,597] WARN [Producer clientId=console-producer] Got error produce response with correlation id 28 on topic-partition test-0, retrying (1 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2019-06-21 12:41:13,597] WARN [Producer clientId=console-producer] Received unknown topic or partition error in produce request on partition test-0. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender)
[2019-06-21 12:41:13,704] WARN [Producer clientId=console-producer] Got error produce response with correlation id 30 on topic-partition test-0, retrying (0 attempts left). Error: UNKNOWN_TOPIC_OR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
[2019-06-21 12:41:13,704] WARN [Producer clientId=console-producer] Received unknown topic or partition error in produce request on partition test-0. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender)
[2019-06-21 12:41:13,808] ERROR Error when sending message to topic test with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
[2019-06-21 12:41:13,809] WARN [Producer clientId=console-producer] Received unknown topic or partition error in produce request on partition test-0. The topic-partition may not exist or the user may not have Describe access to it (org.apache.kafka.clients.producer.internals.Sender)