failed to mount volume /dev/rbd1 [xfs]

3/31/2019

I install kubernetes with 1 master and 3 worker and deploy rook on it and the ceph status in rook-tools shows ok. there is wordpress template in https://github.com/rook/rook.git and when i create it the pod does not created and

#kubectl describe pods wordpress-mysql-b78774f44-m548z -n default
  Type     Reason       Age                  From              Message
  ----     ------       ----                 ----              -------
  Warning  FailedMount  15m (x51 over 128m)  kubelet, ubuntu2  Unable to mount volumes for pod "test-pod-rbd_rook-ceph(15abe007-53a4-11e9-abd9-7c8bca00f216)": timeout expired waiting for volumes to attach or mount for pod "rook-ceph"/"test-pod-rbd". list of unmounted volumes=[data]. list of unattached volumes=[data default-token-8p9br]
  Warning  FailedMount  18s (x72 over 130m)  kubelet, ubuntu2  MountVolume.SetUp failed for volume "pvc-fd3fdbc4-53b7-11e9-abd9-7c8bca00f216" : mount command failed, status: Failure, reason: failed to mount volume /dev/rbd1 [xfs] to /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-fd3fdbc4-53b7-11e9-abd9-7c8bca00f216, error executable file not found in $PATH
#kubectl get events
18h         Warning   FailedMount               pod/wordpress-mysql-b78774f44-m548z                         MountVolume.SetUp failed for volume "pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216" : mount command failed, status: Failure, reason: failed to mount volume /dev/rbd0 [xfs] to /var/lib/kubelet/plugins/ceph.rook.io/rook-ceph-system/mounts/pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216, error executable file not found in $PATH
18h         Warning   FailedMount               pod/wordpress-mysql-b78774f44-m548z                         Unable to mount volumes for pod "wordpress-mysql-b78774f44-m548z_default(e1f6de90-53b6-11e9-abd9-7c8bca00f216)": timeout expired waiting for volumes to attach or mount for pod "default"/"wordpress-mysql-b78774f44-m548z". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-bktfl]
35m         Warning   FailedMount               pod/wordpress-mysql-b78774f44-m548z                         MountVolume.SetUp failed for volume "pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216" : mount command failed, status: Failure, reason: Rook: Error getting RPC client: error connecting to socket /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ceph.rook.io~rook-ceph-system/.rook.sock: dial unix /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ceph.rook.io~rook-ceph-system/.rook.sock: connect: connection refused
6m32s       Warning   FailedMount               pod/wordpress-mysql-b78774f44-m548z                         Unable to mount volumes for pod "wordpress-mysql-b78774f44-m548z_default(e1f6de90-53b6-11e9-abd9-7c8bca00f216)": timeout expired waiting for volumes to attach or mount for pod "default"/"wordpress-mysql-b78774f44-m548z". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-bktfl]
4m17s       Warning   FailedMount               pod/wordpress-mysql-b78774f44-m548z                         MountVolume.SetUp failed for volume "pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume replicapool/pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216: failed to map image replicapool/pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216 cluster rook-ceph. failed to map image replicapool/pvc-e1e758a8-53b6-11e9-abd9-7c8bca00f216: Failed to complete 'rbd': signal: interrupt. . output:

How can i solve it? Is it a bug?

-- yasin lachini
kubernetes
kubernetes-rook

2 Answers

9/6/2019

to resolve this issue, you must change fstype: xfs -> ext4 in your StorageClass

-- Анатолий Кудряшов
Source: StackOverflow

4/1/2019

This looks like there is a problem with pv,pvc. Please run more basics checks, like:

    kubectl get pods,pv,pvc --all-namespaces
    kubectl describe pvc-fd3fdbc4-53b7-11e9-abd9-7c8bca00f216 
    kubectl get pvc pvc-fd3fdbc4-53b7-11e9-abd9-7c8bca00f216 -o yaml
    kubectl get pod wordpress-mysql-b78774f44-m548z -o yaml 
    kubectl -n rook-ceph get all
    kubectl get storageclasses --all-namespaces

You can find more helpful information about troubleshooting techniques and commands when you are working with Rook

Hope this help and please share what you were able to find so we can troubleshoot this.

-- Hanx
Source: StackOverflow