Kubernetes: Cinder volume not working

5/2/2017

I'm trying to fire up sample mysql pod with Cinder PD (https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-cinder-pd).

I have created the volume:

$ cinder show fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc
+------------------------------+--------------------------------------+
|           Property           |                Value                 |
+------------------------------+--------------------------------------+
|         attachments          |                  []                  |
|      availability_zone       |                 nova                 |
|           bootable           |                false                 |
|     consistencygroup_id      |                 None                 |
|          created_at          |      2017-05-02T08:12:32.000000      |
|         description          |                 None                 |
|          encrypted           |                False                 |
|              id              | fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc |
|           metadata           |                  {}                  |
|         multiattach          |                False                 |
|             name             |             k8s-test-pd              |
| os-vol-tenant-attr:tenant_id |   4a7d3b5645254117af1d36c4eb5201a5   |
|      replication_status      |               disabled               |
|             size             |                  2                   |
|         snapshot_id          |                 None                 |
|         source_volid         |                 None                 |
|            status            |              available               |
|          updated_at          |      2017-05-02T08:12:33.000000      |
|           user_id            |   11c0fd1d2a3a475aaeb31f1f637c2f91   |
|         volume_type          |                 None                 |
+------------------------------+--------------------------------------+

and modified the volumeID in sample yaml.

Pod does not come up and describe pod operation returns

Events:
FirstSeen   LastSeen    Count   From                SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----                -------------   --------    ------      -------
  13m       13m     1   {default-scheduler }                Normal      Scheduled   Successfully assigned mysql to mm-kube-slave-1
  11m       16s     6   {kubelet mm-kube-slave-1}           Warning     FailedMount Unable to mount volumes for pod "mysql_default(9ca3450e-2f32-11e7-b455-fa163ed6d9bd)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]
  11m       16s     6   {kubelet mm-kube-slave-1}           Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]

And syslog on my only minion (where the pod is scheduled) says:

May  2 12:37:29 mm-kube-slave-1 kubelet[1979]: I0502 12:37:29.573805    1979 reconciler.go:231] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc" (spec.Name: "mysql-persistent-storage") pod "9ca3450e-2f32-11e7-b455-fa163ed6d9bd" (UID: "9ca3450e-2f32-11e7-b455-fa163ed6d9bd")
May  2 12:37:29 mm-kube-slave-1 kubelet[1979]: E0502 12:37:29.577353    1979 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc\"" failed. No retries permitted until 2017-05-02 12:39:29.577306663 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc" (spec.Name: "mysql-persistent-storage") pod "9ca3450e-2f32-11e7-b455-fa163ed6d9bd" (UID: "9ca3450e-2f32-11e7-b455-fa163ed6d9bd") is not yet attached according to node status
May  2 12:37:42 mm-kube-slave-1 kubelet[1979]: E0502 12:37:42.602080    1979 kubelet.go:1549] Unable to mount volumes for pod "mysql_default(9ca3450e-2f32-11e7-b455-fa163ed6d9bd)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]; skipping pod
May  2 12:37:42 mm-kube-slave-1 kubelet[1979]: E0502 12:37:42.602697    1979 pod_workers.go:182] Error syncing pod 9ca3450e-2f32-11e7-b455-fa163ed6d9bd ("mysql_default(9ca3450e-2f32-11e7-b455-fa163ed6d9bd)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql". list of unattached/unmounted volumes=[mysql-persistent-storage]

Any hint what is wrong? I have set up k8s 1.6.2 using kubeadm with Openstack provider (correct openstack configuration present and --cloud-provider parameters passed to the kubelet. Ubuntu Xenial with Docker 17.04.0-ce.

-- Misko
kubernetes
openstack
openstack-cinder

1 Answer

5/3/2017

This could be a bug in attach/detach controller. Can you open an issue in kube repo with this information ?

-- Humble
Source: StackOverflow