I build kafka cluster on kubernetes following this guilde https://github.com/kubernetes/contrib/tree/master/statefulsets/kafka, it works well for sending/producing message from kubernetes. when I try to expose the kafka cluster using NodePort as follows, kafka clients try to consumer/produce the message from the address 10.xx.xx.xx:30092
will be failed:
apiVersion: v1
kind: Service
metadata:
name: kafka-nodeport
labels:
app: kafka
spec:
type: NodePort
ports:
- port: 9092
nodePort: 30092
name: server
selector:
app: kafka
Why that happened and How to expose the kafka service?
The Kafka Statefulsets require headless service for accessing the brokers. You can change the headless service to Type=NodePort and setting the externalTrafficPolicy=Local. This bypasses the internal load balancing of a Service and traffic destined to a specific node on that node port will only work if a Kafka pod is on that node.
apiVersion: v1
kind: Service
metadata:
name: kafka-nodeport
labels:
app: kafka
spec:
externaTrafficPolicy: Local
type: NodePort
ports:
- port: 9092
nodePort: 30092
name: server
selector:
app: kafka
For example, we have two nodes nodeA and nodeB, nodeB is running a kafka pod. nodeA:30092 will not connect but nodeB:30092 will connect to the kafka pod running on nodeB.
Hope this helps.
You have to do two things :
1) create either a service resource of type NodePort (like you did) or an Ingress resource.
2) Set the two properties below in Kafka : ADVERTISED_HOST and ADVERTISED_PORT.
Kafka will register to Zookeeper with the ip adress of the node, inside containers network it's the internal ip adress that it used so we have to set thoses properties to tell Kafka to register them instead.
Hopes that's help!