I've been trying to get MFA working with kubectl to secure access to the EKS masters in AWS. Docs seem to hint that it's possible but I'm running into problems and I can't figure it out.

Without anything special, I can connect to my EKS cluster:

kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   2d

If I add MFA to the AWS CLI, following these posts:

How to use MFA with AWS CLI?

https://docs.aws.amazon.com/cli/latest/userguide/cli-roles.html

I set up my local credentials like so:

[default]
aws_access_key_id = **********************
aws_secret_access_key = **********************
region = us-west-2

[default_role]
mfa_serial = arn:aws:iam::1234567890:mfa/spanktar
role_arn = arn:aws:iam::1234567890:role/test_assumeRole
source_profile = default

...then I can see that the AWS CLI has MFA:

aws sts get-caller-identity --profile default_role
Enter MFA code for arn:aws:iam::1234567890:mfa/spanktar: 123456
{
    "UserId": "**********************:botocore-session-1234567890",
    "Account": "1234567890",
    "Arn": "arn:aws:sts::1234567890:assumed-role/test_assumeRole/botocore-session-1234567890"
}

Then I can verify:

aws sts assume-role --role-arn arn:aws:iam::1234567890:role/test_assumeRole --role-session-name default_role
{
    "Credentials": {
        "AccessKeyId": "**********************",
        "SecretAccessKey": "**********************",
        "SessionToken": "FOO",
        "Expiration": "2018-10-11T21:19:20Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "**********************:default_role",
        "Arn": "arn:aws:sts::1234567890:assumed-role/test_assumeRole/default_role"
    }
}

kubectl still works as normal because it isn't using the profile yet:

kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   2d

Then I set up k8s to use the profile, following this documentation

users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: "AWS_PROFILE"
        value: "default_role"
      args:
        - "token"
        - "-i"
        - "test-eks-cluster"

So now when I try kubectl it asks for MFA, but it can never be satisfied:

kubectl get svc
Assume Role MFA token code: 123456
Assume Role MFA token code: 123456
could not get token: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code.
    status code: 403, request id: 123456-cd93-11e8-80a5-1234567898765
E1011 13:22:07.385200   28191 exec.go:230] refreshing credentials: exec: exit status 1
No resources found.
error: You must be logged in to the server (Unauthorized)

The AWS Role test_assumeRole looks like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::1234567890:user/spanktar"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The related policy test_assumePolicy looks like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::1234567890:role/test_assumeRole"
        }
    ]
}

So, the questions:

• Do I need to attach any of this to the IAM user? The docs seem to suggest not.
• What am I missing here to get this to operate? It seems almost wired up correctly.