pod with pvc stuck on container creating

11/19/2019

My overall issue is that my pod which has a PVC is stuck on container-creating after it was deleted. My guess why, is because of the following:

So, I have a pod with a mounted PVC. I did a: kubectl exec -it "name" bash navigated to the path of the mounted PVC and wanted to create a tar gzip file of several directories. The reason was because I wanted to copy the folders to local, but they were quite big. Anyways, managed to create the tar file, but someone else released to our dev environment and the pod was killed. After that, when recreating our env, the pod with the PVC that has the tar file is stuck on container creating. Is it because that I created that file on the PVC? Like, based on the warnings it seems like the PVC points to the previous pod?

kubectl get pvc

NAME           STATUS   VOLUME CAPACITY   ACCESS MODES   STORAGECLASS      AGE
graphite-pvc   Bound    xxxx   256Gi      RWO            managed-premium   12

and if I do, i get the following warnings:

kubectl describe pod xxx

Warning FailedAttachVolume 22m (x8 over 24m) attachdetach-controller
AttachVolume.Attach failed for volume "pvc-f65cb358-014b-11ea-b698-000d3a556597" : Attach volume "kubernetes-dynamic-pvc-f65cb358-014b-11ea-b698-000d3a556597" to instance "/subscriptions/1405bf18-bf7d-4a2f-9aa7-25ff73ba58a6/resourceGroups/cie-dev-2-1-eastus/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-dev-nodes-2002/virtualMachines/6" failed with compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status= Code="ConflictingUserInput" Message="Disk '/subscriptions/1405bf18-bf7d-4a2f-9aa7-25ff73ba58a6/resourceGroups/cie-dev-2-1-eastus/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-f65cb358-014b-11ea-b698-000d3a556597' cannot be attached as the disk is already owned by VM '/subscriptions/1405bf18-bf7d-4a2f-9aa7-25ff73ba58a6/resourceGroups/cie-dev-2-1-eastus/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-dev-nodes-2002/virtualMachines/k8s-dev-nodes-2002_111'."

and

Warning FailedMount 48s (x13 over 28m) kubelet, k8s-dev-nodes-2002000006 Unable to mount volumes for pod "xxxx": timeout expired waiting for volumes to attach or mount for pod "xxxxxx". list of unmounted volumes=[pvc_name]. list of unattached volumes=[pvc_name default-token-6tmkm]

So, first, do you think it has any correlation with the fact that I was inside the PVC and created a file, when the pod was killed, or is it pure coincidence (cannot be, right?).

-- Christian Hjelmslund
azure-disk
kubernetes
kubernetes-pvc

0 Answers