I have a question about the behavior of Kubernetes when dealing with Attach a volume on a new node after a reschedule of a pod.
A common behavior we have in our cluster is:
A node n1 becomes unavailable
A pod A with Volume v1 is rescheduled on node n2
Volume v1 is being detached from node n1, this will take a few seconds
kubelet on node n2 tries to Attach Volume v1 to pod A
Because Volume v1 is not yet detached from node n1, the Attach call fails with:
Sep 27 11:43:24 node n2 kubelet-wrapper[787]: E0927 11:43:24.794713 787 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/cinder/volume_v1_id\"" failed. No retries permitted until 2018-09-27 11:43:25.294659022 +0000 UTC m=+1120541.835247469 (durationBeforeRetry 500ms). Error: "AttachVolume.Attach failed for volume \"volume v1\" (UniqueName: \"kubernetes.io/cinder/volume_2_id\") from node \"node n2\" : disk volume_v2_id path /dev/vdc is attached to a different instance (pod node n1)"After this Attach error, the kubelet will eternally try to mount Volume v1 (which will fail because the Volume is not attached)
Sep 27 11:43:26 node n2 kubelet-wrapper[787]: E0927 11:43:26.870106 787 attacher.go:240] Error: could not find attached Cinder disk "volume_v1_id" (path: ""): <nil>My question is: Why does k8s not try to Attach again before trying to Mount?
The issue here is that when the detach is being done quickly enough we do not have any issues, but if the detach is not yet done when the Attach is called by the kubelet, we are stuck.
When digging into the code it seems that the behavior is WaitForAttachAndMount. This will: 1/ Try Attach 2/ whatever the result of the attach, loop on Try Mount.
Should the expected behavior be 1/ loop on Try Attach 2/ If at some point Attach is a success, loop on Try Mount?
This question is related to https://github.com/kubernetes/kubernetes/issues/69158
My take, it depends:
It makes sense to continue trying to Attach indefinitely rather than fail and try to mount indefinitely if you want to make the volume provider (which could be EBS, Cinder, GCP, Ceph, etc) responsible for responding to an attach API. It may be that the provider is doing some maintenance and the APIs are failing temporarily. This is if you want to make your systems more automated.
It makes sense to Attach -> fail and mount indefinitely if for reason you want to let the user manually attach the volume and have a subsequent mount succeed. In my opinion, this should be made an option and 1. should be the default.