I got this error when I created the eks cluster using the root from the eks console. I recreated the eks cluster using an IAM user and use the access keys to update the aws configure. It worked. Now you can add additional IAM users to issue kubectl commands.

I had the same problem, my AWS credentials for CLI change frequently. These steps fixed the problem:

export AWS_ACCESS_KEY_ID="***************"
export AWS_SECRET_ACCESS_KEY="*************"
export AWS_SESSION_TOKEN="************************"

I commented out the last two lines of the config file

# - "-r"
# - "arn:aws:iam::**********:role/**********"

and it worked though I have no idea why

Also, make sure your users are in the aws-auth k8s ConfigMap:

https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

You need to create the cluster under the same IAM profile that you are accessing it from via AWS cli.

Said in another way, inside ~/.aws/credentials, the profile that is accessing kubectl must match exactly the same IAM that was used to create the cluster.

My recommendation is to use AWS cli to create your clusters as creating from the GUI may be more confusing than helpful. The Getting Started guide is your best bet to get up and running.

I had the same problem . It's likely that you are using a root account. It appears root accounts are blocked from assuming the required roles. This error can sometimes be cloaked if you are using expired keys.

If you are using eksctl to manage your aws eks deployments you can add the user to the config map with one command:

eksctl create iamidentitymapping --cluster <cluster-name> --arn arn:aws:iam::<id>:user/<user-name> --group system:masters --username ops-user

This happens also to me with local environment on minikube, independently of EKS. My problem is related to this issue: https://github.com/kubernetes/kubernetes/issues/76774

The solution i adopted is to remove the cache directories of kubectl: rm -rf ~/.kube/{cache,http-cache}. I guess is the only workaround at the time of writing.

In my case it is the AWS profile issue, be sure to use aws sts get-caller-identity to verify the IAM user.

I am sure issue is resolved but I will be putting more information here so if any other people are still facing the issue related to any of the below setup then they might not waste time like me and use the steps.

When we create the EKS cluster by any method via CloudFormation/CLI/EKSCTL the IAM role/user who created the cluster will automatically binded to the default kubernetes RBAC API group system:masters (https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) and in this way creator of the cluster will get the admin access to the cluster. Although we can always give the access to other IAM user/role using the aws-auth file but for that we must have to use the IAM user/role who created the cluster.

To verify the role/user for the EKS cluster we can search for the CreateCluster" Api call on cloudtrail and it will tell us the creator of the cluster in the sessionIssuer section for field arn (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).

When we create the cluster using the IAM role or IAM user, setting up the access for the EKS cluster will become little tricky when we created the cluster using the role compare to user.

I will put the steps we can follow for each different method while setting up the access to EKS cluster.

Scenario-1: Cluster was Created using the IAM user (For example "eks-user")

Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity

$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

Scenario-2: Cluster was Created using the IAM Role (For example "eks-role")

Mainly there are four different way to setup the access via cli when cluster was created via IAM role.

1. Setting up the role directly in kubeconfig file.

In this case we do not have to make any assume role api call via cli manually, before running kubectl command because that will be automatically done by aws/aws-iam-authenticator set in the kube config file.

Lets say now we are trying to setup the access for the user eks-user the first make sure that user does have permission to assume the role eks-role

Add the assume role permission to the eks-user

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::xxxxxxxxxxx:role/eks-role"
        }
    ]
}

Edit the trust relationship on the role so that it will allow the eks-user to assume the role.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity. Important thing to remember it should show us the IAM user ARN not the IAM assumed ROLE ARN.

$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name --role-arn arn:aws:iam::xxxxxxxxxxx:user/eks-role

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      - --role
      - arn:aws:iam::xxxxxxx:role/eks-role
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

2. If you have setup the AWS profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) on CLI and if you want to use that with the kube config.

Confirm that profile is set properly so that it can use the credentials for the eks-user

 $ cat ~/.aws/config
[default]
output = json
region = us-east-1
[eks]
output = json
region = us-east-1
[profile adminrole]
role_arn = arn:aws:iam::############:role/eks-role
source_profile = eks

 $ cat ~/.aws/credentials
[default]
aws_access_key_id = xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[eks]
aws_access_key_id =  xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Once this profile configuration is done please confirm that profile configuration is fine by running the command aws sts get-caller-identity --profile eks

$ aws sts get-caller-identity --profile eks
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command with the profile and please make sure we are not using the role here.

aws eks update-kubeconfig --name devel --profile eks

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws
      env:
      - name: AWS_PROFILE
        value: eks

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

3. Assume the role by any other way, For example we can attach the IAM role to the instance directly.

If role is directly attached to the instance profile then we can follow the similar steps as we followed while setting up the access for IAM user in Scenario-1

Verify that we have attached the correct role to EC2 instance and as this instance profile will come into least precedence, this step will also verify that there are no any other credentials setup on the instnace.

[ec2-user@ip-xx-xxx-xx-252 ~]$ aws sts get-caller-identity
{
    "Account": "xxxxxxxxxxxx",
    "UserId": "xxxxxxxxxxxxxxxxxxxxx:i-xxxxxxxxxxx",
    "Arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/eks-role/i-xxxxxxxxxxx"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

4. Manually assuming the IAM role via aws sts assume-role command.

Assume the role eks-role manually by running the cli command.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxxxxxx:role/eks-role --role-session-name test

{
    "AssumedRoleUser": {
        "AssumedRoleId": "xxxxxxxxxxxxxxxxxxxx:test",
        "Arn": "arn:aws:sts::xxxxxxxxxxx:assumed-role/eks-role/test"
    },
    "Credentials": {
        "SecretAccessKey": "xxxxxxxxxx",
        "SessionToken": xxxxxxxxxxx",
        "Expiration": "xxxxxxxxx",
        "AccessKeyId": "xxxxxxxxxx"
    }
}

After that set the required environment variable using the value from above output so that we can use the correct credentials generated from the session.

export AWS_ACCESS_KEY_ID=xxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx
export AWS_SESSION_TOKEN=xxxxxxxxxx

After that verify that we assumed the IAM role by running the command aws sts get-caller-identity.

$ aws sts get-caller-identity { "Account": "xxxxxxxxxx", "UserId": "xxxxxxxxxx:test", "Arn": "arn:aws:sts::xxxxxxxxxx:assumed-role/eks-role/test" }

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

NOTE:

I have try to cover major use case here but there might be other use case too where we need to setup the access to the cluster.

Also the above tests are mainly aiming at the first time setup of the EKS cluster and none of the above method is touching the aws-auth configmap yet. But once you have given access to other IAM user/role to EKS cluster via aws-auth (https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) file you can use the same set of commands for those users too as mentioned in above answer.

I just debugged this issue. I have a question. Are you running this on a corporate wifi network? If yes, could you create an EC2 instance and then test if you are able to do kubectl get svc?

Also, try if this command works kubectl get svc --insecure-skip-tls-verify