I have Kubernetes 1.18 with rookio setup, this pod was running for sometime. one of the node went out of Ready status for some reason. I rebooted the node, now its in Ready status.

But Pod stuck on ContainerCreating status. its waiting to mount the rookio PVC.

Pod status

# kgp |grep -v Running
NAME                                                        READY   STATUS              RESTARTS   AGE
redis-slave-0                                               0/1     ContainerCreating   0          14h

PodEvents

Events:
  Type     Reason              Age                      From                     Message
  ----     ------              ----                     ----                     -------
  Warning  FailedMount         23m (x69 over 13h)       kubelet, node05       Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[redis-data config redis-tmp-conf default-token-lqpgm health]: timed out waiting for the condition
  Warning  FailedMount         19m (x95 over 13h)       kubelet, node05       Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[health redis-data config redis-tmp-conf default-token-lqpgm]: timed out waiting for the condition
  Warning  FailedMount         14m (x79 over 13h)       kubelet, node05       Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[default-token-lqpgm health redis-data config redis-tmp-conf]: timed out waiting for the condition
  Warning  FailedMount         5m45s (x66 over 13h)     kubelet, node05       Unable to attach or mount volumes: unmounted volumes=[redis-data], unattached volumes=[config redis-tmp-conf default-token-lqpgm health redis-data]: timed out waiting for the condition
  Warning  FailedAttachVolume  2m44s (x101 over 6h32m)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c" : attachdetachment timeout for volume 0001-0009-rook-ceph-0000000000000002-0c4a5173-e8a7-11ea-9bd1-0637030c9151

PVC attach status set to false

 kubectl get volumeattachment |grep -v true  
NAME                                                                   ATTACHER                     PV                                         NODE        ATTACHED   AGE
csi-3424d1bdc5212aeef30e681c9d99df38dd68fdabb47e5f820125c90d54d61d7b   rook-ceph.rbd.csi.ceph.com   pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c   node05   false      14h

I try to move the pod to different node, still same issue.

PV and PVC status

# k describe pv pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c
Name:            pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: rook-ceph.rbd.csi.ceph.com
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    rook-ceph-block
Status:          Bound
Claim:           default/redis-data-redis-slave-0
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        287Mi
Node Affinity:   <none>
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            rook-ceph.rbd.csi.ceph.com
    FSType:            ext4
    VolumeHandle:      0001-0009-rook-ceph-0000000000000002-0c4a5173-e8a7-11ea-9bd1-0637030c9151
    ReadOnly:          false
    VolumeAttributes:      clusterID=rook-ceph
                           imageFeatures=layering
                           imageFormat=2
                           imageName=csi-vol-0c4a5173-e8a7-11ea-9bd1-0637030c9151
                           journalPool=replicapool
                           pool=replicapool
                           radosNamespace=
                           storage.kubernetes.io/csiProvisionerIdentity=1598460149789-8081-rook-ceph.rbd.csi.ceph.com



 k describe pvc redis-data-redis-slave-0
Name:          redis-data-redis-slave-0
Namespace:     default
StorageClass:  rook-ceph-block
Status:        Bound
Volume:        pvc-e854eee7-0a36-4a92-ba61-f9e6e976f64c
Labels:        app=redis
               component=slave
               heritage=Helm
               release=redis
               role=slave
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      287Mi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    redis-slave-0
Events:        <none>

How to fix this mount issue?

Thanks SR