cannot produce messages to kafka when running in kubernetes cluster

7/11/2018

I am trying to run kafka inside kubernetes cluster. I am using the incubator helm chart to deploy kafka and zookeeper, I am deploying with no extra parameters - basically it is kafka cluster of 3 nodes. However when I am trying to connect using my application , it is failing while it was working perfectly when i run it inside docker swarm using the Confluent image. I am getting an error of

2018-07-11 19:25:16 WARN {org.apache.kafka.clients.NetworkClient processDisconnection} - Consumer clientId=consumer-1, groupId=rest-to-kafka-consumer Connection to node -1 could not be established. Broker may not be available.

first i have tried that those will be the properties for the producer

new Properties()
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, EnvVariables.brokerEndpoint.get)
props.put(ProducerConfig.ACKS_CONFIG, "all")
props.put(ProducerConfig.RETRIES_CONFIG, "0")
props.put(ProducerConfig.BATCH_SIZE_CONFIG, "16384")
props.put(ProducerConfig.LINGER_MS_CONFIG, "1")
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "33554432")
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")

where EnvVariables.brokerEndpoint.get is the kafka servicekubectl get svc -n occm NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE occm-kafka ClusterIP 10.31.241.149 <none> 9092/TCP 3d

after that i have changed the configuration to connect using zookeeper - but still i am getting the same error

new Properties()
props.put("zookeeper.connect", EnvVariables.brokerEndpoint.get)
props.put("zookeeper.session.timeout.ms", "100000")
props.put(ProducerConfig.ACKS_CONFIG, "all")
props.put(ProducerConfig.RETRIES_CONFIG, "0")
props.put(ProducerConfig.BATCH_SIZE_CONFIG, "16384")
props.put(ProducerConfig.LINGER_MS_CONFIG, "1")
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, "33554432")
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer")

what do I need to change? using the test client of the sh files works fine

-- li-raz
apache-kafka
kubernetes

0 Answers