k8s - Cinder "0/x nodes are available: x node(s) had volume node affinity conflict"

3/7/2019

I have my own cluster k8s. I'm trying to link the cluster to openstack / cinder.

When I'm creating a PVC, I can a see a PV in k8s and the volume in Openstack. But when I'm linking a pod with the PVC, I have the message k8s - Cinder "0/x nodes are available: x node(s) had volume node affinity conflict".

My yml test:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: classic
provisioner: kubernetes.io/cinder
parameters:
  type: classic

---


kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-infra-consuldata4
  namespace: infra
spec:
  storageClassName: classic
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consul
  namespace: infra
  labels:
    app: consul
spec:
  replicas: 1
  selector:
    matchLabels:
      app: consul
  template:
    metadata:
      labels:
        app: consul
    spec:
      containers:
      - name: consul
        image: consul:1.4.3
        volumeMounts:
        - name: data
          mountPath: /consul
        resources:
          requests:
            cpu: 100m
          limits:
            cpu: 500m
        command: ["consul", "agent", "-server", "-bootstrap", "-ui", "-bind", "0.0.0.0", "-client", "0.0.0.0", "-data-dir", "/consul"]
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: pvc-infra-consuldata4

The result:

kpro describe pvc -n infra
Name:          pvc-infra-consuldata4
Namespace:     infra
StorageClass:  classic
Status:        Bound
Volume:        pvc-76bfdaf1-40bb-11e9-98de-fa163e53311c
Labels:        
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc-infra-consuldata4","namespace":"infra"},"spec":...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age   From                         Message
  ----       ------                 ----  ----                         -------
  Normal     ProvisioningSucceeded  61s   persistentvolume-controller  Successfully provisioned volume pvc-76bfdaf1-40bb-11e9-98de-fa163e53311c using kubernetes.io/cinder
Mounted By:  consul-85684dd7fc-j84v7
kpro describe po -n infra consul-85684dd7fc-j84v7
Name:               consul-85684dd7fc-j84v7
Namespace:          infra
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=consul
                    pod-template-hash=85684dd7fc
Annotations:        <none>
Status:             Pending
IP:                 
Controlled By:      ReplicaSet/consul-85684dd7fc
Containers:
  consul:
    Image:      consul:1.4.3
    Port:       <none>
    Host Port:  <none>
    Command:
      consul
      agent
      -server
      -bootstrap
      -ui
      -bind
      0.0.0.0
      -client
      0.0.0.0
      -data-dir
      /consul
    Limits:
      cpu:  2
    Requests:
      cpu:        500m
    Environment:  <none>
    Mounts:
      /consul from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nxchv (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc-infra-consuldata4
    ReadOnly:   false
  default-token-nxchv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nxchv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  36s (x6 over 2m40s)  default-scheduler  0/6 nodes are available: 6 node(s) had volume node affinity conflict. 

Why K8s successful create the Cinder volume, but it can't schedule the pod ?

-- combivel
kubernetes
openstack-cinder

2 Answers

1/22/2020

Try finding out the nodeAffinity of your persistent volume:

$ kubctl describe pv pvc-76bfdaf1-40bb-11e9-98de-fa163e53311c
Node Affinity:     
  Required Terms:  
    Term 0:        kubernetes.io/hostname in [xxx]

Then try to figure out if xxx matches the node label yyy that your pod is supposed to run on:

$ kubectl get nodes
NAME      STATUS   ROLES               AGE   VERSION
yyy       Ready    worker              8d    v1.15.3

If they don't match, you will have the "x node(s) had volume node affinity conflict" error, and you need to re-create the persistent volume with the correct nodeAffinity configuration.

-- Yuci
Source: StackOverflow

3/7/2019

You have setup provisioner: kubernetes.io/cinder, which based on Kubernetes documentation on Storage Classes - OpenStack Cinder:

Note:

FEATURE STATE: Kubernetes 1.11 deprecated

This internal provisioner of OpenStack is deprecated. Please use the external cloud provider for OpenStack.

Based on OpenStack GitHub you should set provisioner: openstack.org/standalone-cinder

Please check the persistent-volume-provisioning cinder for detailed use and yaml files.

You might be also interested in reading those StackOverflow questions:

Kubernetes Cinder volumes do not mount with cloud-provider=openstack

How to use OpenStack Cinder to create storage class and dynamically provision persistent volume in Kubernetes Cluster

-- Crou
Source: StackOverflow