How to produce to kafka broker running inside container from outside the docker host?

7/2/2018

I am trying to produce to a kafka broker which is running inside the container launched by kubernetes. I am playing with KAFKA_ADVERTISED_LISTENERES and KAFKA_LISTERNERS.

I tried setting these two env variables KAFKA_ADVERTISED_LISTENERES = PLAINTEXT://<host-ip>:9092 and KAFKA_LISTERNERS = PLAINTEXT://0.0.0.0:9092 and ran using docker-compose. And I was able to produce from an application out of the host machine.

But setting these two env-variables in Kubernetes.yml file, I get No broker list available exception.

What am I missing here?

Update:

kafka-pod.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: casb-deployment
  name: kafkaservice
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafkaservice
    spec:
      hostname: kafkaservice
      #hostNetwork: true  # to access docker out side of host container
      containers:
      - name: kafkaservice
        imagePullPolicy: IfNotPresent
        image: wurstmeister/kafka:1.1.0
        env:  # for production 
         - name: KAFKA_ADVERTISED_LISTENERES
           value: "PLAINTEXT://<host-ip>:9092"
         - name: KAFKA_LISTERNERS
           value: "PLAINTEXT://0.0.0.0:9092"
         - name: KAFKA_CREATE_TOPICS
           value: "Topic1:1:1,Topic2:1:1"
         - name: KAFKA_MESSAGE_TIMESTAMP_TYPE
           value: "LogAppendTime"
         - name: KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE
           value: "LogAppendTime"   
         - name: KAFKA_ZOOKEEPER_CONNECT
           value: "zookeeper:2181"
        ports: 
        - name: port9092
          containerPort: 9092

---

apiVersion: v1
kind: Service
metadata:
  namespace: casb-deployment
  name: kafkaservice 
  labels:
    app: kafkaservice
spec:
  selector:
    app: kafkaservice
  ports:
  - name: port9092
    port: 9092
    targetPort: 9092
    protocol: TCP
-- el323
apache-kafka
docker
kubernetes

1 Answer

7/2/2018

I'm assuming you have a Kubernetes service, whose selector links the ingress flow to your Kafka Broker, that is exposing the nodePort (as opposed to clusterIP). https://kubernetes.io/docs/concepts/services-networking/service/

So the kubernetes pod should be reachable through localhost:<nodePort>.

You can also set a Load Balancer in front of your Kubernetes cluster then you can just expose the k8s pods, i.e., allow external ingress.

Then the next step is to just leverage some DNS record so the outbound request produced by your docker-compose-based containers will go to DNS and then come back to your Kubernetes cluster through the load balancer.

-- the_marcelo_r
Source: StackOverflow