unable to deploy debezium on minikube

12/8/2018

I am new to kubernetes, I am trying to integrate kafka with debezium and mysql. i successfully deploy kafka and mysql on minikube , once i deploy the debezium yml on minikube, it got hanged and don't response at all , then i restart the minikube, After running all pod minikube again got hanged. below is my code:

zookeeper service

 apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

zookeeper deployment:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: debezium/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1

kafka service:

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "1"
  type: NodePort

kafka deployemnt:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker1
spec:
  template:
    metadata:
      labels:
        selector: kafka
        app: kafka
        id: "1"
    spec:
      containers:
      - name: kafka
        image: debezium/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "9092"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.39.47
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_CREATE_TOPICS
          value: hello-topic:3:3

MySql-persistance volume:

#application/mysql/mysql-pv.yaml  
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

mysql deployment:

#application/mysql/mysql-deployment.yaml 
# this command is for mysql client   kubectl run -it --rm --image=debezium/example-mysql --restart=Never mysql-client -- mysql -h mysql -pdebezium 
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: extensions/v1beta1  # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: debezium/example-mysql
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          value: debezium
        - name: MYSQL_USER
          value: mysqluser
        - name: MYSQL_PASSWORD
          value: mysqlpw
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

Debezium deployment:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: debezium-connect-source
spec:
  selector:
    matchLabels:
      app: debezium-connect-source
  replicas: 1
  template:
    metadata:
      labels:
        app: debezium-connect-source
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: debezium-connect-source
        image: debezium/connect
        env:
        - name: BOOTSTRAP_SERVERS
          value: kafka-service:9092
        - name: GROUP_ID
          value: "1"
        - name: CONFIG_STORAGE_TOPIC
          value: debezium-connect-source_config
        - name: OFFSET_STORAGE_TOPIC
          value: debezium-connect-source_offset
        ports:
        - containerPort: 8083
          name: dm-c-source

when i deploy the debezium , then problem starts and minikube response like

$ kubectl get pods
Unable to connect to the server: net/http: TLS handshake timeout

OS :Centos
minikube version: v0.30.0

-- Sanjay
apache-kafka
debezium
kubernetes
minikube

2 Answers

12/12/2018

you should set memory limits for the Java-based pods. The older versions of Java would see the whole guest memory as their own and will happily consume it completely - and there are at least three JVMs started.

-- Jiri Pechanec
Source: StackOverflow

12/8/2018

I believe this is happening because of the resource crunch on the VM started by minikube.

By default when you start using minikube start it takes up only 2 CPU and 2GB RAM from your system, and by looking at your deployments (kafka + mysql + debezium) that might not be enough.

You can increase CPU and memory allocated to VM by using minikube start with parameters --cpu and --memory (value should be in MB).

For more info, you should do minikube start -h

I strongly suggest, if you want to setup a heavy deployments you should be using machines with more resources.

Hope this helps.

-- Prafull Ladha
Source: StackOverflow