I install the incubator Kafka chart. Version kafka-0.8.5 as of this writing.

helm install --name kafka \
  --set replicas=1 \
  --set persistence.enabled=false \
  --set zookeeper.replicaCount=1 \
  incubator/kafka

To try this out I run a separate pod as just a bash shell with Kafka cli tools. I'm using the exact same Docker image confluentinc/cp-kafka:4.1.1-2 that the kafka-0 pod is using so that there is a perfect version match between the client and server:

kubectl run shell --rm -i --tty --image confluentinc/cp-kafka:4.1.1-2 -- /bin/bash

Listing topics, publishing messages, getting topic offsets all works perfectly, as shown below. However, when I try to run kafka-console-consumer and see the test record in the topic, it hangs indefinitely. Why?

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-topics --zookeeper kafka-zookeeper:2181 --list

__confluent.support.metrics

root@shell-5c6ddf5d99-tbsvm:/# echo "abcxyz" | /usr/bin/kafka-console-producer --broker-list kafka:9092 --topic test-topic

>[2018-08-07 16:43:26,110] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-topics --zookeeper kafka-zookeeper:2181 --list

__confluent.support.metrics
test-topic

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-run-class kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test-topic --time -1

test-topic:0:1

root@shell-5c6ddf5d99-tbsvm:/# /usr/bin/kafka-console-consumer --bootstrap-server kafka:9092 --from-beginning --topic test-topic

<hangs indefinitely>

FYI, this is a local minikube development cluster with the latest minikube with Kubernetes 1.10.x server-side and 1.10.x kubectl client tools. This is a clean, new minikube, with nothing else running besides kafka, kafka-zookeeper, and my shell pod.

Also, writing a small Java client test app to consume gets a similar result of polling indefinitely with no messages. When my Java client subscribes to test-topic, it never gets notification callbacks of being assigned to the one topic partition.