I am deploying a Jenkins service on K8S as a StatefulSet.

My VolumeMounts looks like :

volumeMounts:
  - name: jenkins-jobs
    mountPath: /srv/jenkins/jobs

And my volumeClaimTemplates section in my StatefulSet declaration looks like:

volumeClaimTemplates:
  - metadata:
      name: jenkins-jobs
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5G
      storageClassName: rbd

I have defined a storageclass rbd and its controller is running in my cluster:

$ kubectl get -n kube-system po | grep rbd
rbd-provisioner-59d8759b85-rzrxr           1/1       Running   0          42m

Now when I spin up my Stateful Set, the pod stays in the below state:

$ kubectl get po -n test-jenkins
NAME               READY     STATUS              RESTARTS   AGE
jenkins-master-0   0/1       ContainerCreating   0          23m

Doing a describe on the Pod yields :

  Normal   SuccessfulMountVolume   2m               kubelet, kubernetes-agent-3  MountVolume.SetUp succeeded for volume "default-token-zcdgk"
  Normal   SuccessfulMountVolume   2m               kubelet, kubernetes-agent-3  MountVolume.SetUp succeeded for volume "local-secrets"
  Normal   SuccessfulAttachVolume  37s              attachdetach-controller      AttachVolume.Attach succeeded for volume "pvc-2bafc440-d259-11e8-953e-02000a1bef39"
  Warning  FailedMount             11s              kubelet, kubernetes-agent-3  Unable to mount volumes for pod "jenkins-master-0_test-jenkins(2bb0d344-d259-11e8-953e-02000a1bef39)": timeout expired waiting for volumes to attach or mount for pod "test-jenkins"/"jenkins-master-0". list of unmounted volumes=[jenkins-jobs]. list of unattached volumes=[jenkins-jobs local-secrets ssh static-jobs update-credentials seed-job renew-token default-token-zcdgk]
  Warning  FailedMount             2s (x4 over 6s)  kubelet, kubernetes-agent-3  MountVolume.WaitForAttach failed for volume "pvc-2bafc440-d259-11e8-953e-02000a1bef39" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: ()

My PVC is created and so is my PV.

$ kubectl get pvc -n test-jenkins
NAME                            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins-jobs-jenkins-master-0   Bound     pvc-2bafc440-d259-11e8-953e-02000a1bef39   4769Mi     RWO            rbd          25m

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                                  STORAGECLASS   REASON    AGE
pvc-2bafc440-d259-11e8-953e-02000a1bef39   4769Mi     RWO            Delete           Bound     test-jenkins/jenkins-jobs-jenkins-master-0   rbd                26m

The logs of my RBD container also show no ERRORs:

I1017 22:08:26.169064       1 controller.go:1068] scheduleOperation[lock-provisiontest-jenkins/jenkins-jobs-jenkins-master-0[2bafc440-d259-11e8-953e-02000a1bef39]]
I1017 22:08:26.186851       1 controller.go:1068] scheduleOperation[lock-provision-test-jenkins/jenkins-jobs-jenkins-master-0[2bafc440-d259-11e8-953e-02000a1bef39]]
I1017 22:08:26.206088       1 controller.go:1068] scheduleOperation[lock-provision-test-jenkins/jenkins-jobs-jenkins-master-0[2bafc440-d259-11e8-953e-02000a1bef39]]
I1017 22:08:26.216657       1 leaderelection.go:156] attempting to acquire leader lease...
I1017 22:08:26.232794       1 leaderelection.go:178] successfully acquired lease to provision for pvc test-jenkins/jenkins-jobs-jenkins-master-0
I1017 22:08:26.232868       1 controller.go:1068] scheduleOperation[provision-test-jenkins/jenkins-jobs-jenkins-master-0[2bafc440-d259-11e8-953e-02000a1bef39]]
I1017 22:08:26.326907       1 provision.go:110] successfully created rbd image "kubernetes-dynamic-pvc-2bbe12a1-d259-11e8-84a1-a6bc2793edca"
I1017 22:08:26.327045       1 controller.go:801] volume "pvc-2bafc440-d259-11e8-953e-02000a1bef39" for claim "test-jenkins/jenkins-jobs-jenkins-master-0" created
I1017 22:08:26.340955       1 controller.go:818] volume "pvc-2bafc440-d259-11e8-953e-02000a1bef39" for claim "test-jenkins/jenkins-jobs-jenkins-master-0" saved
I1017 22:08:26.340981       1 controller.go:854] volume "pvc-2bafc440-d259-11e8-953e-02000a1bef39" provisioned for claim "test-jenkins/jenkins-jobs-jenkins-master-0"
I1017 22:08:28.245193       1 leaderelection.go:198] stopped trying to renew lease to provision for pvc test-jenkins/jenkins-jobs-jenkins-master-0, task succeeded

So based on my describe command, I'm not sure why I can attach the VolumeMount but not mount it. Any idea?