EKS cannot create persistent volume

11/22/2019

I am deploying prometheus which needs persistent volume(i have also tried with other statefulset), but persistent volume is not created and persistent volume clam shows the flowing error after kubectl describe -n {namespace} {pvc-name}.

Type: Warning 

Reason: ProvisioningFailed

From: persistentvolume-controller

Message: (combined from similar events): Failed to provision volume with StorageClass "gp2": error querying for all zones: error listing AWS instances: "UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403, request id: d502ce90-8af0-4292-b872-ca04900d41dc"
kubectl get sc
NAME            PROVISIONER             AGE
gp2 (default)   kubernetes.io/aws-ebs   7d17h
kubectl describe sc gp2

Name:            gp2
IsDefaultClass:  Yes
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"}
,storageclass.kubernetes.io/is-default-class=true
Provisioner:           kubernetes.io/aws-ebs
Parameters:            fsType=ext4,type=gp2
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

K8s versions(aws eks):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-eks-b7174d", GitCommit:"b7174db5ee0e30c94a0b9899c20ac980c0850fc8", GitTreeState:"clean", BuildDate:"2019-10-18T17:56:01Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

helm version

version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}
-- mad_boy
aws-eks
eks
kubernetes
kubernetes-pvc

2 Answers

11/22/2019

I solved this problem by adding AmazonEKSClusterPolicy and AmazonEKSServicePolicy to the eks cluster role.enter image description here

-- mad_boy
Source: StackOverflow

11/22/2019

Check your IAM policies, and ensure that you are using the correct access keys. Also check IAM role has permission to provision dynamic storage

-- P Ekambaram
Source: StackOverflow