We are running an EKS cluster with nodes created with a node group based on AWS autoscaling group.

The cluster is created using eksctl.

The node works ok and after a while, it fails to communicate to the Kubernetes API.

Digging into the problem we found the following. The aws-iam-authenticator returns tokens that fail when used to access the API.

Finally, we found the following situation, we tried to verify the token using the same aws-iam-authenticator, but the validations fails with the message that the token is expired.

We check the situation running the following command:

aws-iam-authenticator verify -i preguntados-dev-v1 -t $( aws-iam-authenticator token -i preguntados-dev-v1 | jq -r '.status.token' )

It requests a token, extracts the token value from the returned JSON, and sends it to verify.

The response is (in the problematic node):

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403). Body: {"Error":{"Code":"ExpiredToken","Message":"The security token included in the request is expired","Type":"Sender"},"RequestId":"477f5292-0e87-4734-bd60-74720a3e13a8"}

aws-iam-authenticator version
{"Version":"v0.5.0","Commit":"1cfe2a90f68381eacd7b6dcfa2bf689e76eb8b4b"}