K
Q

Unable to create statefulset and PVC in the same AWS AZ

1/10/2019

Im trying to launch two Cassandra statefulset instances and respective PVC in a cluster created in AWS AZ (Across 3 zones, eu-west-1a, eu-west-1b & eu-west-1c).

I created a node group with the following 2 nodes so as shown these nodes attach in the zones eu-west-1a and eu-west-1b

ip-192-168-47-86.eu-west-1.compute.internal - failure-domain.beta.kubernetes.io/zone=eu-west-1a,node-type=database-only
ip-192-168-3-191.eu-west-1.compute.internal - failure-domain.beta.kubernetes.io/zone=eu-west-1b,node-type=database-only

When I launch the Cassandra instances (using Helm) only one instance starts. The other instance shows the error,

0/4 nodes are available: 2 node(s) didn't match node selector, 2 node(s) had no available volume zone.

The PVCs for these instances are bounded,

kubectl get pvc -n storage -o wide
NAME                            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cassandra-data-cc-cassandra-0   Bound     pvc-81e30224-14c5-11e9-aa4e-06d38251f8aa   10Gi       RWO            gp2            4m
cassandra-data-cc-cassandra-1   Bound     pvc-abd30868-14c5-11e9-aa4e-06d38251f8aa   10Gi       RWO            gp2            3m

However, the PVs show that they are in zones eu-west-1b & eu-west-1c

kubectl get pv -n storage --show-labels
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                   STORAGECLASS   REASON    AGE       LABELS
pvc-81e30224-14c5-11e9-aa4e-06d38251f8aa   10Gi       RWO            Delete           Bound     storage/cassandra-data-cc-cassandra-0   gp2                      7m        failure-domain.beta.kubernetes.io/region=eu-west-1,failure-domain.beta.kubernetes.io/zone=eu-west-1b
pvc-abd30868-14c5-11e9-aa4e-06d38251f8aa   10Gi       RWO            Delete           Bound     storage/cassandra-data-cc-cassandra-1   gp2                      6m        failure-domain.beta.kubernetes.io/region=eu-west-1,failure-domain.beta.kubernetes.io/zone=eu-west-1c

I have tried adding the following topology to the StorageClass to no avail,

allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - eu-west-1a
    - eu-west-1b

But despite of this I can still see the PVs in the zones, eu-west-1b & eu-west-1c.

Using K8 1.11.

Any other possible fixes ?

-- nixgadget
aws-eks
kubernetes
kubernetes-pvc
kubernetes-statefulset

Similar Questions

Unable to fetch broker ID using kubectl command
Deny access to route outside of kubernetes cluster
Azure AKS LoadBalancer not properly working
Does Liveness probe verify certificate if scheme is set to HTTPS in kubernetes?
Send logs to s3 - Official Airflow Helm Chart + KubernetesExecutor

1 Answer

1/10/2019

Looking at https://v1-11.docs.kubernetes.io/docs/concepts/storage/storage-classes/ allowedTopologies doesnt exist.

So I used zones: eu-west-1a, eu-west-1b in the StorageClass which seems to have worked.

provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  zones: eu-west-1a, eu-west-1b
-- nixgadget
Source: StackOverflow