I have been trying to run kafka/zookeeper on Kubernetes. Using helm charts I am able to install zookeeper on the cluster. However the ZK pods are stuck in pending state. When I issued describe on one of the pod "didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate." was the reason for scheduling failure. But when I issue describe on PVC , I am getting "waiting for first consumer to be created before binding". I tried to re-spawn the whole cluster but the result is same. Trying to use https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/ as guide.

Can someone please guide me here ?

kubectl get pods -n zoo-keeper

kubectl get pods -n zoo-keeper
NAME                         READY   STATUS    RESTARTS   AGE
zoo-keeper-zk-0              0/1     Pending   0          20m
zoo-keeper-zk-1              0/1     Pending   0          20m
zoo-keeper-zk-2             0/1     Pending   0          20m

kubectl get sc

kubectl get sc
NAME            PROVISIONER                    AGE
local-storage   kubernetes.io/no-provisioner   25m

kubectl describe sc

kubectl describe  sc
Name:            local-storage
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"local-storage"},"provisioner":"kubernetes.io/no-provisioner","volumeBindingMode":"WaitForFirstConsumer"}

Provisioner:           kubernetes.io/no-provisioner
Parameters:            <none>
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

kubectl describe pod foob-zookeeper-0 -n zoo-keeper

ubuntu@kmaster:~$ kubectl describe pod foob-zookeeper-0 -n zoo-keeper
Name:               foob-zookeeper-0
Namespace:          zoo-keeper
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=foob-zookeeper
                    app.kubernetes.io/instance=data-coord
                    app.kubernetes.io/managed-by=Tiller
                    app.kubernetes.io/name=foob-zookeeper
                    app.kubernetes.io/version=foob-zookeeper-9.1.0-15
                    controller-revision-hash=foob-zookeeper-5321f8ff5
                    release=data-coord
                    statefulset.kubernetes.io/pod-name=foob-zookeeper-0
Annotations:        foobar.com/product-name: zoo-keeper ZK
                    foobar.com/product-revision: ABC
Status:             Pending
IP:
Controlled By:      StatefulSet/foob-zookeeper
Containers:
  foob-zookeeper:
    Image:       repo.data.foobar.se/latest/zookeeper-3.4.10:1.6.0-15
    Ports:       2181/TCP, 2888/TCP, 3888/TCP, 10007/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Limits:
      cpu:     2
      memory:  4Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Liveness:   exec [zkOk.sh] delay=15s timeout=5s period=10s #success=1 #failure=3
    Readiness:  tcp-socket :2181 delay=15s timeout=5s period=10s #success=1 #failure=3
    Environment:
      ZK_REPLICAS:           3
      ZK_HEAP_SIZE:          1G
      ZK_TICK_TIME:          2000
      ZK_INIT_LIMIT:         10
      ZK_SYNC_LIMIT:         5
      ZK_MAX_CLIENT_CNXNS:   60
      ZK_SNAP_RETAIN_COUNT:  3
      ZK_PURGE_INTERVAL:     1
      ZK_LOG_LEVEL:          INFO
      ZK_CLIENT_PORT:        2181
      ZK_SERVER_PORT:        2888
      ZK_ELECTION_PORT:      3888
      JMXPORT:               10007
    Mounts:
      /var/lib/zookeeper from datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nfcfx (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-foob-zookeeper-0
    ReadOnly:   false
  default-token-nfcfx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nfcfx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  69s (x4 over 3m50s)  default-scheduler  0/2 nodes are available: 1 node(s) didn't find available persistent volumes to bind, 1 node(s) had taints that the pod didn't tolerate.

kubectl get pv

ubuntu@kmaster:~$ kubectl get  pv
NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS    REASON   AGE
local-pv   50Gi       RWO            Retain           Available           local-storage            10m
ubuntu@kmaster:~$

kubectl get pvc local-claim

ubuntu@kmaster:~$ kubectl get  pvc local-claim
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
local-claim   Pending                                      local-storage   8m9s
ubuntu@kmaster:~$

kubectl describe pvc local-claim

ubuntu@kmaster:~$ kubectl describe pvc local-claim
Name:          local-claim
Namespace:     default
StorageClass:  local-storage
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                    From                         Message
  ----       ------                ----                   ----                         -------
  Normal     WaitForFirstConsumer  2m3s (x26 over 7m51s)  persistentvolume-controller  waiting for first consumer to be created before binding
Mounted By:  <none>

MY PV files:

cat create-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 50Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/kafka-mount
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kmaster

cat pvc.yml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-claim
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 50Gi