K
Q

Exposing Kafka cluster in Kubernetes using LoadBalancer service

3/27/2019

Suppose if I have 3 node Kafka cluster setup. Then how do I expose it outside a cloud using Load Balancer service? I have read reference material but have a few doubts.

Say for example below is a service for a broker

apiVersion: v1 
  kind: Service metadata: 
  name: kafka-0 
  annotations: dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com 
  spec: 
    externalTrafficPolicy: Local 
    type: LoadBalancer 
    ports: 
      - port: 9092 
      name: outside 
      targetPort: 9092 
    selector: app: kafka kafka-pod-id: "0"
  1. What is port and targetPort?
  2. Do I setup LoadBalancer service for each of the brokers?
  3. Do these multiple brokers get mapped to single public IP address of cloud LB?
  4. How does a service outside k8s/cloud access individual broker? By using public-ip:port? or by using kafka-<pod-id>.kafka.my.company.com:port?. Also which port is used here? port or targetPort?
  5. How do I specify this configuration in Kafka broker's Advertised.listeners property? As port can be different for services inside k8s cluster and outside it.

Please help.

-- Shades88
apache-kafka
kubernetes

Similar Questions

Minikube: Path to Root Certificate
equivalent of "minikube ssh" with docker-for-desktop Kubernetes node
Kubernetes config file from host to container?
Is there a way to make kustomize merge headings in a yaml file instead of completely replacing that heading with the overlay yaml

1 Answer

5/21/2019

Based on the information you provided I will try give you some answers, eventually give some advise.

1) port: is the port number which makes a service visible to other services running within the same K8s cluster. In other words, in case a service wants to invoke another service running within the same Kubernetes cluster, it will be able to do so using port specified against port in the service spec file.

targetPort: is the port on the POD where the service is running. Your application needs to be listening for network requests on this port for the service to work.

2/3) Each Broker should be exposed as LoadBalancer and be configured as headless service for internal communication. There should be one addiational LoadBalancer with external ip for external connection.

Example of Service

apiVersion: v1
kind: Service
metadata:
  name: kafka-0
  annotations: dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    pod-name: kafka-0
  type: LoadBalancer

4) You have to use kafka-<pod-id>.kafka.my.company.com:port

5) It should be set to the external addres so that clients can connect to it. This article might help with understanding.

Similar case was on Github, it might help you also - https://github.com/kow3ns/kubernetes-kafka/issues/3

In addition, You could also think about Ingress - https://tothepoint.group/blog/accessing-kafka-on-google-kubernetes-engine-from-the-outside-world/

-- PjoterS
Source: StackOverflow