How to resize an ECK cluster

7/15/2020

I have an elasticsearch cluster that has the storage field set to 10Gi, I want to resize this cluster (for testing purposes to 15Gi). However, after changing the storage value from 10Gi to 15Gi I can see that the cluster still did not resize and the generated PVC is still set to 10Gi.

From what I can tell the aws-ebs storage https://kubernetes.io/docs/concepts/storage/storage-classes/ allows for volume expansion when the field allowVolumeExpansion is true. But even when I have this, the volume is never expanded when I change that storage value

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: elasticsearch-storage
  namespace: test
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: test
spec:
  version: 7.4.2
  spec:
    http:
      tls:
        certificate:
          secretName: es-cert
  nodeSets:
  - name: default
    count: 3
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
        annotations:
          volume.beta.kubernetes.io/storage-class: elasticsearch-storage
      spec:
        accessModes:
        - ReadWriteOnce
        storageClassName: elasticsearch-storage
        resources:
          requests:
            storage: 15Gi
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
      xpack.security.authc.realms:
        native:
          native1: 
            order: 1
---
-- Kay
elasticsearch
kubernetes

1 Answer

7/15/2020

Technically it should work but your Kubernetes cluster might not be able to connect to the AWS API to expand the volume. Did you check the actual EBS volume on the EC2 console or AWS CLI? You can debug this issue by looking at the kube-controller-manager and cloud-controller manager logs.

My guess is that there is some type of permission issue that from your K8s cluster that cannot talk to your AWS/EC2 API.

If you are running EKS, make sure that the IAM cluster role that you are using has permissions for EC2/EBS. You can check the control plane logs (kube-controller-manager, kube-apiserver, cloud-controller-manager, etc) on CloudWatch.

EDIT:

The Elasticsearch operator uses StatefulSets and as of this date Volume expansion is not supported on StatefulSets.

-- Rico
Source: StackOverflow