K
Q

What node field means in k8s volume.attachments?

8/12/2020

I have the following Volume: 

Name:         pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
Namespace:    rook-ceph-system
Labels:       <none>
Annotations:  <none>
API Version:  rook.io/v1alpha2
Attachments:
  Cluster Name:   rook-ceph
  Mount Dir:      /var/lib/kubelet/pods/72fd4f89-5110-49b7-8d88-87488b58695c/volumes/ceph.rook.io~rook-ceph-system/pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
  Node:           node-6.xyz.com
  Pod Name:       dev-cockroachdb-0
  Pod Namespace:  x-namespace
  Read Only:      false
Kind:             Volume
Metadata:
  Creation Timestamp:  2020-08-12T17:13:51Z
  Generation:          6
  Resource Version:    638003207
  Self Link:           /apis/rook.io/v1alpha2/namespaces/rook-ceph-system/volumes/pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
  UID:                 db0a9491-95fe-49cd-8160-89031847d636
Events:                <none>

For the pod dev-cockroachdb-0 I'm getting the following error:

MountVolume.SetUp failed for volume "pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738 for pod x-namespace/dev-cockroachdb-0. Volume is already attached by pod x-namespace/dev-cockroachdb-0. Status Pending

And the pod x-namespace/dev-cockroachdb-0 is currently scheduled to node-5.xyz.com.

So, as you can see the pod itself is in a different node than the VolumeAttachment. node-6.xyz.com vs node-5.xyz.com.

Questions:

  • Does Node in Volume.Attachments point to the node on which the pod (to which the volume is attached) is located? (So if the volume is attached to a pod on node NodeA, then the value of node field for the volume attachment will be NodeA)
  • May this error happen because of the failure to correctly detach the volume on some node?
-- V&#252;sal
kubernetes
kubernetes-rook
persistent-volumes
rook-storage

Similar Questions

chown: changing ownership of '/data/db': Operation not permitted
kubectl get cs prompt Error from server (Forbidden)
How to bootstrap RBAC privileges when bringing up a GKE cluster with Terraform
TCP Ingress with Istio 0.8 and v1alpha3 Gateway
Mounting k8 permanent volume fails silently

1 Answer

8/13/2020

You didn't mention it but it looks like you have a Rook volume. Perhaps a pvc-clone, something like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc-clone
spec:
  storageClassName: rook-ceph-block
  dataSource:
    name: rbd-pvc
    kind: PersistentVolumeClaim
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Does Node in Volume.Attachments point to the node on which the pod (to which the volume is attached) is located? (So if the volume is attached to a pod on node NodeA, then the value of node field for the volume attachment will be NodeA)

Not really, your Pod may be trying to start on node-5.xyz.com, but the volume could be attached to another node in your case it's node-6.xyz.com.

May this error happen because of the failure to correctly detach the volume on some node?

Yes, this error may happen if maybe you had another pod running in node-6.xyz.com that terminated and failed to detach.

Keep in mind that all of the above is considering that your volume has accessModes: ReadWriteOnce. It looks like you are using Ceph with Rook and in that case, you can also use accessModes: ReadWriteMany which would essentially allow you to attach the volume to Kubernetes nodes.

✌️

-- Rico
Source: StackOverflow