From Step 1: Create Your Amazon Cluster

When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator (with system:master permissions. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl.

As you have discovered you can only access the cluster with the same user/role that created the EKS cluster in the first place.

There is a way to add additional roles to the cluster after creation by editing the aws-auth ConfigMap that has been created.

Add User Role

By editing the aws-auth ConfigMap you can add different levels of access based on the role of the user.

First you MUST have the "system:node:{{EC2PrivateDNSName}}" user

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: <ARN of instance role (not instance profile)>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

This is required for Kubernetes to even work, giving the nodes the ability to join the cluster. The "ARN of instance role" is the role that includes the required policies AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly etc.

Below that add your role

   - rolearn: arn:aws:iam::[hidden]:role/ci_deployer
     username: ci-deployer
     groups:
       - system:masters

The 'username' can actually be set to about anything. It appears to only be important if there are custom roles and bindings added to your EKS cluster.

Also, use the command 'aws sts get-caller-identity' to validate the environment/shell and the AWS credentials are properly configured. When correctly configured 'get-caller-identity' should return the same role ARN specified in aws-auth.

Can kubectl work from an assumed role from AWS

Yes, it can work. A good way to troubleshoot it is to run from the same command line where you are running kubectl:

$ aws sts get-caller-identity

You can see the Arn for the role (or user) and then make sure there's a trust relationship in IAM between that and the role that you specify here in your kubeconfig:

command: aws-iam-authenticator
args:
   - "token"
   - "-i"
   - "<cluster-name>"
   - "-r"
   - "<role-you-want-to-assume-arn>"

or with the newer option:

command: aws
args:
- eks
- get-token
- --cluster-name
- <cluster-name>
- --role
- <role-you-want-to-assume-arn>

Note that if you are using aws eks update-kubeconfig you can pass in the --role-arn flag to generate the above in your kubeconfig.

In your case, some things that you can look at:

• The credential environment variables are not set in your CI?:

  AWS_ACCESS_KEY_ID
  AWS_SECRET_ACCESS_KEY
  

• Your ~/.aws/credentials file is not populated correctly in your CI. With something like this:

  [default]
  aws_access_key_id = xxxx
  aws_secret_access_key = xxxx

• Generally, the environment variables take precedence so it could be that you could have different credentials altogether in those environment variables too.

• It could also be the AWS_PROFILE env variable or the AWS_PROFILE config in ~/.kube/config

  users:
  - name: aws
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1alpha1
        command: aws-iam-authenticator
        args:
          - "token"
          - "-i"
          - "<cluster-name>"
          - "-r"
          - "<role-arn>"
        env:
          - name: AWS_PROFILE <== is this value set
            value: "<aws-profile>"

• Is the profile set correctly under ~/.aws/config?