I am unable to hit a kubernetes service on port 9092. The same service exposes a port 29092 as well, and I am able to access that. The trouble is, I am running Kafka inside of Kube. I used docker stack to deploy resources from a docker-compose file onto kube. This has resulted in the creation of services like below:

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                          AGE
broker                      ClusterIP      None             <none>        55555/TCP                        65m
broker-published            LoadBalancer   10.109.34.43     localhost     9092:30229/TCP,29092:32557/TCP   65m

I am now trying to access the Kafka broker via localhost:9092 and I am unable to do so. I can access 29092 without any trouble. I can only access 9092 if I port forward kubectl port-forward service/broker-published 9092:9092

When I describe the service as a yaml, I get the below:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-12-31T01:14:18Z"
  labels:
    com.docker.service.id: local-kafka-broker
    com.docker.service.name: broker
    com.docker.stack.namespace: local-kafka
  name: broker-published
  namespace: payments
  ownerReferences:
  - apiVersion: compose.docker.com/v1alpha3
    blockOwnerDeletion: true
    controller: true
    kind: Stack
    name: local-kafka
    uid: dd6045a4-2b6a-11ea-9dab-025000000001
  resourceVersion: "428377"
  selfLink: /api/v1/namespaces/payments/services/broker-published
  uid: de1b9498-2b6a-11ea-898e-025000000001
spec:
  clusterIP: 10.109.34.43
  externalTrafficPolicy: Cluster
  ports:
  - name: 9092-tcp
    nodePort: 30229
    port: 9092
    protocol: TCP
    targetPort: 9092
  - name: 29092-tcp
    nodePort: 32557
    port: 29092
    protocol: TCP
    targetPort: 29092
  selector:
    com.docker.service.id: local-kafka-broker
    com.docker.service.name: broker
    com.docker.stack.namespace: local-kafka
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: localhost

In answer to a few comments raised below. I am using docker stack to deploy a docker compose yml into my local kubernetes cluster. Here's the docker compose file:

---
version: '3.7'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.1
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-enterprise-kafka:5.3.1
    hostname: broker
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

  schema-registry:
    image: confluentinc/cp-schema-registry:5.3.1
    hostname: schema-registry
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'

  kafka-cat:
    image: edenhill/kafkacat:1.5.0
    entrypoint:
      - sh
      - -c
      - "exec tail -f /dev/null"
    environment:
      schemaRegistry: "http://schema-registry:8081"
      broker: 'broker:29092'
      zookeeper: 'zookeeper:2181'

  connect:
    image: debezium/connect:0.10
    hostname: connect
    ports:
      - 8083:8083
    environment:
      - BOOTSTRAP_SERVERS=broker:29092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses
      - KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
      - VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
      - INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
      - CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081

  sqlserver:
    image: microsoft/mssql-server-linux:2017-CU9-GDR2
    ports:
      - 1433:1433
    environment:
      - ACCEPT_EULA=Y
      - MSSQL_PID=Standard
      - SA_PASSWORD=Password!
      - MSSQL_AGENT_ENABLED=true


  control-center:
    image: confluentinc/cp-enterprise-control-center:5.3.1
    hostname: control-center
    ports:
      - "9021:9021"
    environment:
      CONTROL_CENTER_BOOTSTRAP_SERVERS: 'broker:29092'
      CONTROL_CENTER_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      CONTROL_CENTER_CONNECT_CLUSTER: 'connect:8083'
      CONTROL_CENTER_KSQL_ADVERTISED_URL: "http://localhost:8088"
      CONTROL_CENTER_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      CONTROL_CENTER_REPLICATION_FACTOR: 1
      CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
      CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1
      CONFLUENT_METRICS_TOPIC_REPLICATION: 1
      PORT: 9021

I am following this blog post and have configured my advertised listeners accordingly - https://rmoff.net/2018/08/02/kafka-listeners-explained/

The command I use to deploy the above onto my local kube cluster -

docker stack deploy --orchestrator=kubernetes --namespace payments -c docker-compose.yml local-kafka

I am trying to connect to the broker by using kafkacat running on mac. I have also tried from some consumer code that I have written and I get a timeout unless I do a port forward.

So this command does not work:

kafkacat -b localhost:9092 -L

unless I first do a

kubectl port-forward service/broker-published 9092:9092