K
Q

Question

Kafka deployment on minikube

6/20/2017

Hi i am new to kubernets, i am using minikube single node cluster for local development and testing.

Host: Ubuntu 16.04 LTS.
Minikube: Virtual box running minikube cluster

My requirement is i need to deploy kafka and zookeeper on minikube and should be used to produce or consume messages. I followed this link and successfully deployed it on minikube its details are below

$ kubectl get services
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
kafka-service   10.0.0.15    <pending>     9092:30244/TCP               46m
kubernetes      10.0.0.1     <none>        443/TCP                      53m
zoo1            10.0.0.43    <none>        2181/TCP,2888/TCP,3888/TCP   50m
zoo2            10.0.0.226   <none>        2181/TCP,2888/TCP,3888/TCP   50m
zoo3            10.0.0.6     <none>        2181/TCP,2888/TCP,3888/TCP   50m

$ kubectl get pods
NAME                                      READY     STATUS    RESTARTS   AGE
kafka-deployment-3583985961-f2301         1/1       Running   0          48m
zookeeper-deployment-1-1598963595-vgx1l   1/1       Running   0          52m
zookeeper-deployment-2-2038841231-tdsff   1/1       Running   0          52m
zookeeper-deployment-3-2478718867-5vjcj   1/1       Running   0          52m

$ kubectl describe service kafka-service
Name:           kafka-service
Namespace:      default
Labels:         app=kafka
Annotations:    <none>
Selector:       app=kafka
Type:           LoadBalancer
IP:             10.0.0.15
Port:           kafka-port  9092/TCP
NodePort:       kafka-port  30244/TCP
Endpoints:      172.17.0.7:9092
Session Affinity:   None
Events:         <none>

and i have set KAFKA_ADVERTISED_HOST_NAME to minikube ip(192.168.99.100). Now for message producer i am using $cat textfile.log | kafkacat -b $(minikube ip):30244 -t mytopic its not publishing the message giving below message

% Auto-selecting Producer mode (use -P or -C to override)
% Delivery failed for message: Local: Message timed out

can any one help how to publish and consume message.

-- user1184777

apache-kafka

kafkacat

kubectl

kubernetes

minikube

4 Answers

8/21/2017

Going off the previous answer and your comment here is very basic sample code for a Kafka service with type=Nodeport.

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
  type: NodePort

-- artur513

Source: StackOverflow

5/9/2020

I followed as shown below.

$ git clone https://github.com/d1egoaz/minikube-kafka-cluster

$ cd minikube-kafka-cluster

$ kubectl apply -f 00-namespace/

$ kubectl apply -f 01-zookeeper/

$ kubectl apply -f 02-kafka/

$ kubectl apply -f 03-yahoo-kafka-manager/

$ kubectl get svc -n kafka-ca1 (get the port of kafka manager like 31445)

$ minikube ip

Put http://minikube-ip:port in browser to get ui page of kafka manager.

source https://technology.amis.nl/2019/03/24/running-apache-kafka-on-minikube/

-- Shree Prakash

Source: StackOverflow

2/18/2018

I know that this is quite an old post. Were you able to resolve and run kafka + zookeeper within minikube? I was able to run a simple single cluster kafka and zookeeper deployment successfully using minikube v0.17.1 and produce and consume messages using kafkacat producer and consumer respectively. I was able to run these successfully on uBuntu and Mac OSX. The deployment and service yamls are as below:

zookeeper-deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper
  name: zookeeper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - image: wurstmeister/zookeeper
        imagePullPolicy: IfNotPresent
        name: zookeeper
        ports:
        - containerPort: 2181

zookeeper-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-service
  name: zookeeper-service
spec:
  type: NodePort
  ports:
  - name: zookeeper-port
    port: 2181
    nodePort: 30181
    targetPort: 2181
  selector:
    app: zookeeper

kafka-deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kafka
  name: kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: "192.168.99.100"
        - name: KAFKA_ADVERTISED_PORT
          value: "30092"
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 192.168.99.100:30181
        - name: KAFKA_CREATE_TOPICS
          value: "test-topic:1:1"
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        name: kafka
        ports:
        - containerPort: 9092

kafka-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  ports:
  - name: kafka-port
    port: 9092
    nodePort: 30092
    targetPort: 9092
  selector:
    app: kafka

You can test your deployment by installing kafkacat client and running the following commands on separate terminal windows:

echo "Am I receiving this message?" | kafkacat -P -b 192.168.99.100:30092 -t test-topic

kafkacat -C -b 192.168.99.100:30092 -t test-topic

% Reached end of topic test-topic [0] at offset 0

Am I receiving this message?

I was able to successfully run this on minikube versions v0.17.1 and v0.19.0. If you want to run this on minikube versions v0.21.1 and v0.23.0, please refer to my reply to the post here: Kafka inaccessible once inside Kubernetes/Minikube

Thanks.

-- Abhilash Nair

Source: StackOverflow

6/20/2017

You used a service with type = LoadBalancer which is used for cloud provider (you can see the service waiting for an external ip address .. pending state ... which will never happen). In your case you should try with NodePort.

-- ppatierno

Source: StackOverflow

KQ

Kafka deployment on minikube

Similar Questions

4 Answers

K
Q