attachable flexVolume on GKE not working

7/12/2018

I'm in the process of writing my own flexVolume driver on a 1.10.2-gke.3 cluster and ran into troubles getting it to work in attachable mode.

The driver works in non-attachable mode but then ReadWriteOnce doesn't enforce mounting on only one node at a time which is a requirement for my driver and the ultimate goal I'm trying to achieve here.

I made the driver attachable by returning the capability in the init call: {"status": "Success", "capabilities": {"attach": true}}

Then, when creating a pod that references the volume with this driver, only getvolumename gets called. It doesn't really matter what I return here or if I implement the call at all. This is also the only invocation of the driver, i.e. never gets called for attach etc.

The logs indicate that the volume is not registered as in use by the node:

kubelet[1910]: W0712 flexVolume driver a3cloud/glusterloop: using default GetVolumeName for volume a3cloud-system-test
kubelet[1910]: I0712 operationExecutor.VerifyControllerAttachedVolume started for volume "a3cloud-system-test" (UniqueName: "flexvolume-a3cloud/glusterloop/a3cloud-system-test") pod "test-56c867dd4d-75hwc" (UID: "4f633bb6-85b1-11e8-adb3-42010a9c0fc8")
kubelet[1910]: E0712 Operation for "\"flexvolume-a3cloud/glusterloop/a3cloud-system-test\"" failed. No retries permitted until 2018-07-12 08:57:24.360177039 +0000 UTC m=+64585.959480766 (durationBeforeRetry 2m2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"a3cloud-system-test\" (UniqueName: \"flexvolume-a3cloud/glusterloop/a3cloud-system-test\") pod \"test-56c867dd4d-75hwc\" (UID: \"4f633bb6-85b1-11e8-adb3-42010a9c0fc8\") "

According to this issue comment, as well as the the docs if the kubelet is started with --enable-controller-attach-detach option the driver must also be installed on the master. Looking at the running processes, this option is not set, but may be a default in the meantime?

So my questions are:

  • Is it possible to write an flexVolume driver with attachable support without having it to install on the master?
  • Is it possible to install flexVolume driver on a GKE master?
  • Is there any other way I can ensure that my volume is mounted on only one node at a time (without hacking up my own locking solution)?

Thanks!

-- grasbueschel
google-kubernetes-engine
kubernetes

1 Answer

8/15/2018

In summary, GKE masters are managed by Google, so you can't access the master. It seems it's not possible to write a flexVolume driver with attachable support without having it to install on the master. However, for this question and the other question (mounting volume on only one node at a time) we recommend contacting the GKE team. They are available on Slack: slack.k8s.io.

-- mehdi sharifi
Source: StackOverflow