Unable to attach or mount volumes : timed out waiting for the condition

2/4/2022

While mounting my EBS volume to the kubernetes cluster I was getting this error :

    Warning  FailedMount  64s   kubelet            Unable to attach or mount volumes: unmounted volumes=[ebs-volume], unattached volumes=[ebs-volume kube-api-access-rq86p]: timed out waiting for the condition

Below are my SC, PV, PVC, and Deployment files

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
mountOptions:
  - debug
volumeBindingMode: Immediate

---

kind: PersistentVolume
apiVersion: v1
metadata:
  name: ebs-pv
  labels:
    type: ebs-pv
spec:
  storageClassName: standard
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-0221ed06914dbc8fd
    fsType: ext4

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ebs-pvc
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---

kind: Deployment
metadata:
  labels:
    app: gitea
  name: gitea
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitea
  template:
    metadata:
      labels:
        app: gitea
    spec:
      volumes:
        - name: ebs-volume
          persistentVolumeClaim:
             claimName: ebs-pvc
      containers:
        - image: gitea/gitea:latest
          name: gitea
          volumeMounts:
              - mountPath: "/data"
                name: ebs-volume

This is my PV and PVC which I believe is connected perfectly

    NAME                      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
persistentvolume/ebs-pv   1Gi        RWO            Retain           Bound    default/ebs-pvc   standard                18m

NAME                            STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/ebs-pvc   Bound    ebs-pv   1Gi        RWO            standard       18m

This is my storage class

NAME       PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard   kubernetes.io/aws-ebs   Retain          Immediate           false                  145m

This is my pod description

Name:           gitea-bb86dd6b8-6264h
Namespace:      default
Priority:       0
Node:           worker01/172.31.91.105
Start Time:     Fri, 04 Feb 2022 12:36:15 +0000
Labels:         app=gitea
                pod-template-hash=bb86dd6b8
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/gitea-bb86dd6b8
Containers:
  gitea:
    Container ID:
    Image:          gitea/gitea:latest
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from ebs-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rq86p (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  ebs-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ebs-pvc
    ReadOnly:   false
  kube-api-access-rq86p:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    20m                  default-scheduler  Successfully assigned default/gitea-bb86dd6b8-6264h to worker01
  Warning  FailedMount  4m47s (x2 over 16m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[ebs-volume], unattached volumes=[kube-api-access-rq86p ebs-volume]: timed out waiting for the condition
  Warning  FailedMount  19s (x7 over 18m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[ebs-volume], unattached volumes=[ebs-volume kube-api-access-rq86p]: timed out waiting for the condition

This is my ebs-volume the last one which I have connected to the master node on which I am performing operations right now...

NAME    FSTYPE   LABEL           UUID                                 MOUNTPOINT
loop0   squashfs                                                      /snap/core18/2253
loop1   squashfs                                                      /snap/snapd/14066
loop2   squashfs                                                      /snap/amazon-ssm-agent/4046
xvda
└─xvda1 ext4     cloudimg-rootfs c1ce24a2-4987-4450-ae15-62eb028ff1cd /
xvdf    ext4                     36609bbf-3248-41f1-84c3-777eb1d6f364

The cluster I have created manually on the AWS ubuntu18 instances, there are 2 worker nodes and 1 master node all on Ubuntu18 instances running on AWS.

Below are the commands which I have used to create the EBS volume.

aws ec2 create-volume --availability-zone=us-east-1c --size=10 --volume-type=gp2
aws ec2 attach-volume --device /dev/xvdf --instance-id <MASTER INSTANCE ID> --volume-id <MY VOLUME ID>
sudo mkfs -t ext4 /dev/xvdf

After this the container was successfully created and attached, so I don't think there will be a problem in this part.

I have not done one thing which I don't know if it is necessary or not is the below part

The cluster also needs to have the flag --cloud-provider=aws enabled on the kubelet, api-server, and the controller-manager during the cluster’s creation

This thing I found on one of the blogs but at that moment my cluster was already set-up so I didn't do it but if it is a problem then please notify me and also please give some guidance about how to do it.

I have used Flannel as my network plugin while creating the cluster.

I don't think I left out any information but if there is something additional you want to know please ask.

Thank you in advance!

-- XANDER_015
amazon-ec2
amazon-web-services
kubernetes
kubernetes-pvc
persistent-volumes

1 Answer

2/4/2022

This is my ebs-volume the last one which I have connected to the master node...

Pod that wish to mount this volume must run on the same node as the PV currently attached. Given the scenario you described; it is currently mounted on your Ubuntu based master node therefore you need to run pod on this node in order to mount it. Otherwise, you need to release it from the master node (detach from the underlying EC2) and re-deploy your PVC/PV/Pod so that they can settle down on a worker node instead of master node.

-- gohm&#39;c
Source: StackOverflow