I am facing issues while mounting pvc on pods with openebs installed on bare-metal kubernetes cluster created with RKE.
PVC's should be mounted on pods without issues.
Pods unable to mount PVC's:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m9s (x23 over 2m45s) default-scheduler pod has unbound PersistentVolumeClaims (repeated 4 times)
Normal Scheduled 2m8s default-scheduler Successfully assigned default/minio-deployment-64d7c79464-966jr to 192.168.1.21
Normal SuccessfulAttachVolume 2m8s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff"
Warning FailedMount 84s (x4 over 102s) kubelet, 192.168.1.21 MountVolume.WaitForAttach failed for volume "pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to sendtargets to portal 10.43.227.122:3260 output: iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: cannot make connection to 10.43.227.122: Connection refused
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: No portals found
, err exit status 21
Warning FailedMount 24s (x4 over 80s) kubelet, 192.168.1.21 MountVolume.WaitForAttach failed for volume "pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2016-09.com.openebs.jiva:pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff, portal: 10.43.227.122,3260].
iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation)
iscsiadm: Could not log into all portals
Logging in to [iface: default, target: iqn.2016-09.com.openebs.jiva:pvc-63cf6c92-ec99-11e8-85c9-b06ebfd124ff, portal: 10.43.227.122,3260] (multiple)
(exit status 12)
Warning FailedMount 2s kubelet, 192.168.1.21 Unable to mount volumes for pod "minio-deployment-64d7c79464-966jr_default(640263d0-ec99-11e8-85c9-b06ebfd124ff)": timeout expired waiting for volumes to attach or mount for pod "default"/"minio-deployment-64d7c79464-966jr". list of unmounted volumes=[storage]. list of unattached volumes=[storage default-token-9n8pn]
openebs-standalone
kubectl get pvc
:
root@an4:/home/rke-k8s# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
docker-private-registry-docker-registry Bound pvc-58cf63c1-ec95-11e8-9b5d-2cfda16d3cfd 10Gi RWO openebs-standalone 22m
Update
When I tried the sample minio deployment, here's what I have observed:
openebs-standard
.Any reason for this? It is on-prem cluster deployment.
This is the issue with openebs and already been opened with team. Fix is still pending, you can track the issue here:
https://github.com/openebs/openebs/issues/1688
There is step by step instruction how to debug the issue. Hope this helps.
Well, this issue was documented in the troubleshooting guide - https://docs.openebs.io/docs/next/tsgiscsi.html