I am trying to mount a persistent volume on pods (via a deployment).
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- image: ...
volumeMounts:
- mountPath: /app/folder
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: volume-claim
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
However, the pod stays in "ContainerCreating" status and the events show the following error message.
Unable to mount volumes for pod "podname": timeout expired waiting for volumes to attach or mount for pod "namespace"/"podname". list of unmounted volumes=[volume]. list of unattached volumes=[volume]
I verified that the persistent volume claim is ok and bound to a persistent volume.
What am I missing here?
If doing this in a cloud provider, the storageClass object will create the respective volume for your persistent volume claim.
If you are trying to do this locally on minikube or in a self managed kubernetes cluster, you need to manually create the storageClass that will provide the volumes for you, or create it manually like this example:
apiVersion: v1
kind: PersistentVolume
metadata:
name: task-pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
The hostPath variable will mount this data in the current pod node.
When you create a PVC
without specifying a PV
or type of StorageClass
in GKE clusters it will fall back to default option:
StorageClass: standard
Provisioner: kubernetes.io/gce-pd
Type: pd-standard
Please take a look on official documentation: Cloud.google.com: Kubernetes engine persistent volumes
There could be a lot of circumstances that can produce error message encountered.
As it's unknown how many replicas are in your deployment as well as number of nodes and how pods were scheduled on those nodes, I've tried to reproduce your issue and I encountered the same error with following steps (GKE cluster was freshly created to prevent any other dependencies that might affect the behavior).
Steps:
replicas > 1
Below is example YAML
definition of a PVC
the same as yours:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: volume-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
After applying above definition please check if it created successfully. You can do it by using below commands:
$ kubectl get pvc volume-claim
$ kubectl get pv
$ kubectl describe pvc volume-claim
$ kubectl get pvc volume-claim -o yaml
replicas > 1
Below is example YAML
definition of deployment with volumeMounts
and replicas
> 1:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ubuntu-deployment
spec:
selector:
matchLabels:
app: ubuntu
replicas: 10 # amount of pods must be > 1
template:
metadata:
labels:
app: ubuntu
spec:
containers:
- name: ubuntu
image: ubuntu
command:
- sleep
- "infinity"
volumeMounts:
- mountPath: /app/folder
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: volume-claim
Apply it and wait for a while.
You can check the state of pods with below command:
$ kubectl get pods -o wide
Output of above command:
NAME READY STATUS RESTARTS AGE IP NODE
ubuntu-deployment-2q64z 0/1 ContainerCreating 0 4m27s <none> gke-node-1
ubuntu-deployment-4tjp2 1/1 Running 0 4m27s 10.56.1.14 gke-node-2
ubuntu-deployment-5tn8x 0/1 ContainerCreating 0 4m27s <none> gke-node-1
ubuntu-deployment-5tn9m 0/1 ContainerCreating 0 4m27s <none> gke-node-3
ubuntu-deployment-6vkwf 0/1 ContainerCreating 0 4m27s <none> gke-node-1
ubuntu-deployment-9p45q 1/1 Running 0 4m27s 10.56.1.12 gke-node-2
ubuntu-deployment-lfh7g 0/1 ContainerCreating 0 4m27s <none> gke-node-3
ubuntu-deployment-qxwmq 1/1 Running 0 4m27s 10.56.1.13 gke-node-2
ubuntu-deployment-r7k2k 0/1 ContainerCreating 0 4m27s <none> gke-node-3
ubuntu-deployment-rnr72 0/1 ContainerCreating 0 4m27s <none> gke-node-3
Take a look on above output:
Running
stateContainerCreating
stateAll of the Running
pods are located on the same gke-node-2
You can get more detailed information why pods are in ContainerCreating
state by:
$ kubectl describe pod NAME_OF_POD_WITH_CC_STATE
The Events
part in above command shows:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned default/ubuntu-deployment-2q64z to gke-node-1
Warning FailedAttachVolume 14m attachdetach-controller Multi-Attach error for volume "pvc-7d756147-6434-11ea-a666-42010a9c0058" Volume is already used by pod(s) ubuntu-deployment-qxwmq, ubuntu-deployment-9p45q, ubuntu-deployment-4tjp2
Warning FailedMount 92s (x6 over 12m) kubelet, gke-node-1 Unable to mount volumes for pod "ubuntu-deployment-2q64z_default(9dc28e95-6434-11ea-a666-42010a9c0058)": timeout expired waiting for volumes to attach or mount for pod "default"/"ubuntu-deployment-2q64z". list of unmounted volumes=[volume]. list of unattached volumes=[volume default-token-dnvnj]
Pod cannot pass ContainerCreating
state because of failed mounting of a volume
. Mentioned volume
is already used by other pods on a different node.
ReadWriteOnce: The Volume can be mounted as read-write by a single node.
Please take a look at: Cloud.google.com: Access modes of persistent volumes.
There is detailed answer on topic of access mode: Stackoverflow.com: Why can you set multiple accessmodes on a persistent volume
As it's unknown what you are trying to achieve please take a look on comparison between Deployments and Statefulsets: Cloud.google.com: Persistent Volume: Deployments vs statefulsets.