My kops configuration has the feature gates and policies as follows (notice that I added a test policy of EC2:* just to make sure the masters and nodes had all the permissions):
spec:
additionalPolicies:
master: |
[
{
"Effect": "Allow",
"Action": ["ec2:*"
],
"Resource": ["*"]
}
]
node: |
[
{
"Effect": "Allow",
"Action": ["ec2:*"
],
"Resource": ["*"]
}
]
and
kubeAPIServer:
featureGates:
ExpandPersistentVolumes: "true"
kubeControllerManager:
featureGates:
ExpandPersistentVolumes: "true"
kubelet:
featureGates:
ExpandPersistentVolumes: "true"
I created a PVC as follows:
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: kek-pvc-2
namespace: kekdev
labels:
app: kek
storage: persistent
annotations:
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
storageClassName: kek-storage-class
StorageClass looks like this:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: kek-storage-class
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
parameters:
type: gp2
fsType: ext4
encrypted: "true"
allowVolumeExpansion: true
k8s properly created the volume in AWS. Then I proceeded to edit the PVC storage request and increment it to, say, 7Gi. I could see in the AWS console how it got resized, optimized and available again. However, in k8s, it doesn't seem to update or do anything. It still shows as bound and with the initial 4Gi size. It has no pods attached whatsoever. If I look at the resource template in the k8s dashboard, I see this:
"conditions": [
{
"type": "Resizing",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2018-07-27T10:02:33Z"
}
]
It looks like it's stuck in resizing state, but it's already resized.
kube logs say:
I0727 09:54:03.546372 1 operation_generator.go:1195] ExpandVolume
succeeded for volume kekdev/kek-pvc-2
I0727 09:54:03.558268 1 operation_generator.go:1207]
ExpandVolume.UpdatePV succeeded for volume kekdev/kek-pvc-2
I0727 09:54:32.909753 1 operation_generator.go:1195] ExpandVolume
succeeded for volume kekdev/kek-pvc-2
I0727 09:54:32.912113 1 operation_generator.go:1207]
ExpandVolume.UpdatePV succeeded for volume kekdev/kek-pvc-2
Any ideas on what could be going on?
kubectl version output:
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Many thanks in advance for any inputs on this.