PVC volume mount on Azure kubernetes takes over an hour

8/24/2017

I have tectonic kubernetes cluster installed on Azure. It's made from tectonic-installer GH repo, from master (commit 0a7a1edb0a2eec8f3fb9e1e612a8ef1fd890c332).

> kubectl version                                                                       

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3+coreos.0", GitCommit:"42de91f04e456f7625941a6c4aaedaa69708be1b", GitTreeState:"clean", BuildDate:"2017-08-07T19:44:31Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

On the cluster I created storage class, PVC and pod as in: https://gist.github.com/mwieczorek/28b7c779555d236a9756cb94109d6695

But the pod cannot start. When I run:

kubectl describe pod mypod

I get in events:

FailedMount     Unable to mount volumes for pod "mypod_default(afc68bee-88cb-11e7-a44f-000d3a28f26a)": 
timeout expired waiting for volumes to attach/mount for pod "default"/"mypod". list of unattached/unmounted volumes=[mypd]

In kubelet logs (https://gist.github.com/mwieczorek/900db1e10971a39942cba07e202f3c50) I see:

Error: Volume not attached according to node status for volume "pvc-61a8dc6a-88cb-11e7-ad19-000d3a28f2d3" 
(UniqueName: "kubernetes.io/azure-disk//subscriptions/abc/resourceGroups/tectonic-cluster-mwtest/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-61a8dc6a-88cb-11e7-ad19-000d3a28f2d3") pod "mypod" (UID: "afc68bee-88cb-11e7-a44f-000d3a28f26a")

When I create PVC - new disc on Azure is created. And after creating pod - I see on the azure portal that the disc is attached to worker VM where the pod is scheduled.

> fdisk -l

shows:

Disk /dev/sdc: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

I found similar issue on GH ( kubernetes/kubernetes/issues/50150) but I have cluster built from master so it's not the udev rules (I checked - file /etc/udev/rules.d/66-azure-storage.rules exists)


Does anybody knows if it's a bug (maybe know issue)?

Or am I doing something wrong?

Also: how can I troubleshoot that further?

-- mwieczorek
azure
kubernetes

2 Answers

10/25/2017

In your case, "kubectl describe pod-name" does not provide suffiecient info, you need to provide k8s contoller manager logs for troubleshooting

Get the controller manager logs on master:

#get the "CONTAINER ID" of "/hyperkube controlle"
docker ps -a | grep "hyperkube controlle" | awk -F ' ' '{print $1}'
#get controller manager logs
docker logs "CONTAINER ID" > "CONTAINER ID".log 2>&1 &
-- andy zhang
Source: StackOverflow

8/25/2017

I had test in lab, use your yaml file to create pod, after one hour, it still show pending.

root@k8s-master-ED3DFF55-0:~# kubectl get pod
NAME          READY     STATUS    RESTARTS   AGE
mypod         0/1       Pending   0          1h
task-pv-pod   1/1       Running   0          2h

We can use this yaml file to create pod:

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mypvc
  namespace: kube-public
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Output:

root@k8s-master-ED3DFF55-0:~# kubectl get pvc --namespace=kube-public
NAME      STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mypvc     Bound     pvc-1b097337-8960-11e7-82fc-000d3a191e6a   100Gi      RWO           default        3h

Pod:

kind: Pod
apiVersion: v1
metadata:
  name: task-pv-pod
spec:

  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
       claimName: task-pv-claim

  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
      - mountPath: "/usr/share/nginx/html"
        name: task-pv-storage

Output:

root@k8s-master-ED3DFF55-0:~# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
task-pv-pod   1/1       Running   0          3h

As a workaround, we can use default as the storageclass.

In Azure, there are managed disk and unmanaged disk. if your nodes are use managed disk, two storage classes will be created to provide access to create Kubernetes persistent volumes using Azure managed disks.

They are managed-premium and managed-standard and map to Standard_LRS and Premium_LRS managed disk types respectively.

If your nodes are use non-managed disk, the default storage class will be used if persistent volume resources don't specify a storage class as part of the resource definition.

The default storage class uses non-managed blob storage and will provision the blob within an existing storage account present in the resource group or provision a new storage account.

Non-managed persistent volume types are available on all VM sizes.

More information about managed disk and non-managed disk, please refer to this link.

Here is the test result:

root@k8s-master-ED3DFF55-0:~# kubectl get pvc --namespace=default
NAME            STATUS    VOLUME                                     CAPACITY   ACCESSMODES   STORAGECLASS       AGE
shared          Pending                                                                       standard-managed   2h
shared1         Pending                                                                       managed-standard   15m
shared12        Pending                                                                       standard-managed   14m
shared123       Bound     pvc-a379ced4-897c-11e7-82fc-000d3a191e6a   2Gi        RWO           default            12m
task-pv-claim   Bound     pvc-3cefd456-8961-11e7-82fc-000d3a191e6a   3Gi        RWO           default            3h

Update: Here is my K8s agent's unmanaged disk: enter image description here

-- Jason Ye
Source: StackOverflow