Unable to mount volumes for pod "metadata-api-local": timeout expired waiting for volumes to attach or mount for pod "metadata"/"metadata-api-local"

9/18/2019

I have a failing service with Kubernetes, it seems that service doesnt want to mount volume.

Unable to mount volumes for pod "metadata-api-local": timeout expired waiting for volumes to attach or mount for pod "metadata"/"metadata-api-local". list of unmounted volumes=[metadata-api-claim]. list of unattached volumes=[metadata-api-claim default-token-8lqmp]

Here is the log:

➜  metadata_api git:(develop) ✗ kubectl describe pod -n metadata metadata-api-local-f5bddb8f7-clmwq
Name:           metadata-api-local-f5bddb8f7-clmwq
Namespace:      metadata
Priority:       0
Node:           minikube/192.168.0.85
Start Time:     Wed, 18 Sep 2019 16:59:02 +0200
Labels:         app=metadata-api-local
                pod-template-hash=f5bddb8f7
Annotations:    <none>
Status:         Pending
IP:             
Controlled By:  ReplicaSet/metadata-api-local-f5bddb8f7
Containers:
  metadata-api-local:
    Container ID:   
    Image:          metadata_api:local
    Image ID:       
    Port:           18000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      metadata-env  Secret  Optional: false
    Environment:    <none>
    Mounts:
      /var/lib/nodered-peer from metadata-api-claim (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8lqmp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  metadata-api-claim:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  metadata-api-claim
    ReadOnly:   false
  default-token-8lqmp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8lqmp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    14m                default-scheduler  Successfully assigned metadata/metadata-api-local-f5bddb8f7-clmwq to minikube
  Warning  FailedMount  47s (x6 over 12m)  kubelet, minikube  Unable to mount volumes for pod "metadata-api-local-f5bddb8f7-clmwq_metadata(94cbb26c-4907-4512-950a-29a25ad1ef20)": timeout expired waiting for volumes to attach or mount for pod "metadata"/"metadata-api-local-f5bddb8f7-clmwq". list of unmounted volumes=[metadata-api-claim]. list of unattached volumes=[metadata-api-claim default-token-8lqmp]

Here is my metadata_pvc.yml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: metadata-api-pv
  namespace: metadata
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    namespace: metadata
    name: metadata-api-claim
  hostPath:
    path: /data/metadata-api
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: metadata-api-claim
  namespace: metadata
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: metadata-postgres-volume
  namespace: metadata
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  claimRef:
    namespace: metadata
    name: metadata-postgres-claim
  hostPath:
    path: /data/metadata-postgres
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: metadata-postgres-claim
  namespace: metadata
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

When I list pv, I get:

➜  metadata_api git:(develop) ✗ kubectl get pv                                     
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                              STORAGECLASS   REASON   AGE
metadata-api-pv            1Gi        RWO            Retain           Available   metadata/metadata-api-claim                                12m
metadata-postgres-volume   1Gi        RWO            Retain           Available   metadata/metadata-postgres-claim                           12m

➜  metadata_api git:(develop) ✗ kubectl get pvc                                    
No resources found.

What is failing ?

-- Juliatzin
docker
kubernetes

1 Answer

9/18/2019

You shouldn't specify claimRef, that field is automatically generated by Kubernetes controllers. Instead you should use storage classes for both your PersistentVolumes and PersistentVolumeClaims as that is the mechanism used to match them. Adding the storageClassName: name field to both your PersistentVolumes and PersistentVolumeClaims should fix your issue.

-- Alassane Ndiaye
Source: StackOverflow