I'm trying my hand on iam roles for services account to secure the autoscaller. But I seem to be missing something. Little precision I'm using terraform to create the cluster.
I followed these documentation:
So I've created a role other than the one for the nodes and applied the policy for the autoscaller to this new role. This part is basic, no issue there.
I also activated the openid provider in terraform:
resource "aws_iam_openid_connect_provider" "example" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = []
url = aws_eks_cluster.eks.identity.0.oidc.0.issuer
}
No issue the cluster is creating itself with no issue.
No I added the annotation to service account for the autoscalling:
---
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::ID:role/terraform-eks-autoscaller
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: cluster-autoscaler
namespace: kube-system
My problem is that it does not seems to works and the pod is still trying to use the new IAM role but still using the node role:
Failed to create AWS Manager: cannot autodiscover ASGs: AccessDenied: User: arn:aws:sts::ID:assumed-role/terraform-eks-node/i-ID is not authorized to perform: autoscaling:DescribeTags
Does someone know what step I'm missing here?
Thanks in advance for the help ;)
So answer is very simple. Your OIDC provider configuration is missing the thumbprint. It is essential for Iam to work correctly. Normally if you create OIDC provider in AWS console that thumbprint gets populated automatically, however it is not the case when you do it through terraform.
I have been caught by this as well so I have written a blog about this that you can find here: https://medium.com/@marcincuber/amazon-eks-with-oidc-provider-iam-roles-for-kubernetes-services-accounts-59015d15cb0c
To solve your issue simply add the following:
9E99A48A9960B14926BB7F3B02E22DA2B0AB7280
The above is the hashed root CA that doesn’t change for another 10+ years and it is the same across all regions. How to acquire it, you can read the blog I added link to above.
Additionally, ensure to use the latest autoscaler version which is matching the version of your kubernetes. Also, try adding security context with fsGroup: 65534. That is the current workaround to make the OIDC work properly for some apps.