I have a python script in our K8s cluster that is run as a k8s Cronjob every few minutes. The script checks the nodes in the cluster and if a node is unhealthy for more than 5 minutes, it terminates the node. To connect to AWS I use Boto3. requirement.txt

boto3==1.16.11
botocore==1.19.11

and the permissions are passed as pod annotations.

Annotations:  iam.amazonaws.com/role: arn:aws:iam::123456789:role/k8s-nodes-monitoring-role

The IAM role has arn:aws:iam::aws:policy/AmazonEC2FullAccess policy and a valid trust policy.

{
   "Version": "2012-10-17",
   "Statement": [
{
  "Effect": "Allow",
  "Principal": {
    "Service": "ec2.amazonaws.com"
  },
  "Action": "sts:AssumeRole"
},
{
  "Sid": "",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::123456789:role/nodes.my-domain.com"
  },
  "Action": "sts:AssumeRole"
}
]
}

The problem that I facing is that on some occasions the script throws NoCredentialsError('Unable to locate credentials') error. This behaviour is not consistent as on most occasions the script has successfully terminates the unhealthy node and I can cross-check it against AWS CloudTrail events. I can see in kub2iam logs that the Get request receives 200 but the Put request receives 403.

ime="2020-12-21T12:50:16Z" level=info msg="GET /latest/meta-data/iam/security- 
credentials/k8s-nodes-monitoring-role (200) took 47918.000000 ns" req.method=GET 
req.path=/latest/meta-data/iam/security-credentials/k8s-nodes-monitoring-role 
req.remote=100.116.203.13 res.duration=47918 res.status=200
time="2020-12-21T12:52:16Z" level=info msg="PUT /latest/api/token (403) took 19352999.000000 
ns" req.method=PUT req.path=/latest/api/token req.remote=100.116.203.14 
res.duration=1.9352999e+07 res.status=40

Any help or idea about how to debug this will be highly appreciated.