Issues expanding a PVC in Kubernetes 1.9.9 / Kops 1.9.2 on AWS EC2 using allowVolumeExpansion

7/27/2018

My kops configuration has the feature gates and policies as follows (notice that I added a test policy of EC2:* just to make sure the masters and nodes had all the permissions):

spec:
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": ["ec2:*"
           ],
          "Resource": ["*"]
        }
      ]
    node: |
      [
        {
          "Effect": "Allow",
          "Action": ["ec2:*"
           ],
          "Resource": ["*"]
        }
      ]

and

  kubeAPIServer:
    featureGates:
      ExpandPersistentVolumes: "true"
  kubeControllerManager:
    featureGates:
      ExpandPersistentVolumes: "true"
  kubelet:
    featureGates:
      ExpandPersistentVolumes: "true"

I created a PVC as follows:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kek-pvc-2
  namespace: kekdev
  labels:
    app: kek
    storage: persistent
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  storageClassName: kek-storage-class

StorageClass looks like this:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: kek-storage-class
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
parameters:
  type: gp2
  fsType: ext4
  encrypted: "true"
allowVolumeExpansion: true

k8s properly created the volume in AWS. Then I proceeded to edit the PVC storage request and increment it to, say, 7Gi. I could see in the AWS console how it got resized, optimized and available again. However, in k8s, it doesn't seem to update or do anything. It still shows as bound and with the initial 4Gi size. It has no pods attached whatsoever. If I look at the resource template in the k8s dashboard, I see this:

"conditions": [
  {
    "type": "Resizing",
    "status": "True",
    "lastProbeTime": null,
    "lastTransitionTime": "2018-07-27T10:02:33Z"
  }
]

It looks like it's stuck in resizing state, but it's already resized.

kube logs say:

I0727 09:54:03.546372       1 operation_generator.go:1195] ExpandVolume 
succeeded for volume kekdev/kek-pvc-2
I0727 09:54:03.558268       1 operation_generator.go:1207] 
ExpandVolume.UpdatePV succeeded for volume kekdev/kek-pvc-2
I0727 09:54:32.909753       1 operation_generator.go:1195] ExpandVolume 
succeeded for volume kekdev/kek-pvc-2
I0727 09:54:32.912113       1 operation_generator.go:1207] 
ExpandVolume.UpdatePV succeeded for volume kekdev/kek-pvc-2

Any ideas on what could be going on?

kubectl version output:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Many thanks in advance for any inputs on this.

--
amazon-ec2
kubernetes

0 Answers