I am trying to setup an AWS ALB Ingress Controller using the IRSA method instead of kube2iam. There is however some lack of documentation so I came to a dead end.
What I did so far:
eksctl utils associate-iam-oidc-provider --cluster devops --approve
Created the proper policy by using the template
Created the IAM service account that will be used by the Ingress Controller and associated the policy
eksctl create iamserviceaccount --name alb-ingress --namespace default --cluster devops --attach-policy-arn arn:aws:iam::112233445566:policy/eks-ingressController-iam-policy-IngressControllerPolicy-1111111111 --approve
kubectl apply -f rbac-role.yaml
ServiceAccount matches the service account I created previously.Everything up to here is deployed fine. Now I try to deploy my Ingress service but I get this error (in the controller logs)
kubebuilder/controller "msg"="Reconciler error" "error"="failed to build LoadBalancer configuration due to failed to get AWS tags. Error: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/eksctl-devops-nodegroup-ng-1-work-NodeInstanceRole-J08FDJHIWPI7/i-000000000000 is not authorized to perform: tag:GetResources\n\tstatus code: 400, request id: 94d614a1-c05d-4b92-8ad6-86b450407f6a" "Controller"="alb-ingress-controller" "Request"={"Namespace":"superset","Name":"superset-ingress"}Obviously the node doesn't have the proper permissions for the ALB creation, and I guess that if I attached my policy to the role stated in the log it would work. But that defeats the whole purpose of doing the IRSA method right?
What I would expect is for the Ingress Controller pod to need the appropriate permissions -by using the service account- to create the ALB and not the Node. Am I missing something here?
According to the documentation need the permission to CRUD an ALB. You could if you wanted to try giving just the ALB driver Pod a role with permissions create the ALB but I have not tested it and I am not sure it matters, if your entire scheduler has been given access to to use the ALB driver/pod to create these objects on AWS.
I am not using EKS's 3.0's cluster creation tool, instead I have my own CFT that I use to create workers due to my orgs additional security requirements.
I have have created and attached the bellow managed policy to workers that need to create ALB's and it just works.
ALBPolicy:
Type: "AWS::IAM::ManagedPolicy"
Properties:
Description: Allows workers to CRUD alb's
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "acm:DescribeCertificate"
- "acm:ListCertificates"
- "acm:GetCertificate"
Resource: "*"
-
Effect: "Allow"
Action:
- "ec2:AuthorizeSecurityGroupIngress"
- "ec2:CreateSecurityGroup"
- "ec2:CreateTags"
- "ec2:DeleteTags"
- "ec2:DeleteSecurityGroup"
- "ec2:DescribeAccountAttributes"
- "ec2:DescribeAddresses"
- "ec2:DescribeInstances"
- "ec2:DescribeInstanceStatus"
- "ec2:DescribeInternetGateways"
- "ec2:DescribeNetworkInterfaces"
- "ec2:DescribeSecurityGroups"
- "ec2:DescribeSubnets"
- "ec2:DescribeTags"
- "ec2:DescribeVpcs"
- "ec2:ModifyInstanceAttribute"
- "ec2:ModifyNetworkInterfaceAttribute"
- "ec2:RevokeSecurityGroupIngress"
Resource: "*"
-
Effect: "Allow"
Action:
- "elasticloadbalancing:AddListenerCertificates"
- "elasticloadbalancing:AddTags"
- "elasticloadbalancing:CreateListener"
- "elasticloadbalancing:CreateLoadBalancer"
- "elasticloadbalancing:CreateRule"
- "elasticloadbalancing:CreateTargetGroup"
- "elasticloadbalancing:DeleteListener"
- "elasticloadbalancing:DeleteLoadBalancer"
- "elasticloadbalancing:DeleteRule"
- "elasticloadbalancing:DeleteTargetGroup"
- "elasticloadbalancing:DeregisterTargets"
- "elasticloadbalancing:DescribeListenerCertificates"
- "elasticloadbalancing:DescribeListeners"
- "elasticloadbalancing:DescribeLoadBalancers"
- "elasticloadbalancing:DescribeLoadBalancerAttributes"
- "elasticloadbalancing:DescribeRules"
- "elasticloadbalancing:DescribeSSLPolicies"
- "elasticloadbalancing:DescribeTags"
- "elasticloadbalancing:DescribeTargetGroups"
- "elasticloadbalancing:DescribeTargetGroupAttributes"
- "elasticloadbalancing:DescribeTargetHealth"
- "elasticloadbalancing:ModifyListener"
- "elasticloadbalancing:ModifyLoadBalancerAttributes"
- "elasticloadbalancing:ModifyRule"
- "elasticloadbalancing:ModifyTargetGroup"
- "elasticloadbalancing:ModifyTargetGroupAttributes"
- "elasticloadbalancing:RegisterTargets"
- "elasticloadbalancing:RemoveListenerCertificates"
- "elasticloadbalancing:RemoveTags"
- "elasticloadbalancing:SetIpAddressType"
- "elasticloadbalancing:SetSecurityGroups"
- "elasticloadbalancing:SetSubnets"
- "elasticloadbalancing:SetWebACL"
Resource: "*"
-
Effect: "Allow"
Action:
- "iam:CreateServiceLinkedRole"
- "iam:GetServerCertificate"
- "iam:ListServerCertificates"
Resource: "*"
-
Effect: "Allow"
Action:
- "cognito-idp:DescribeUserPoolClient"
Resource: "*"
-
Effect: "Allow"
Action:
- "waf-regional:GetWebACLForResource"
- "waf-regional:GetWebACL"
- "waf-regional:AssociateWebACL"
- "waf-regional:DisassociateWebACL"
Resource: "*"
-
Effect: "Allow"
Action:
- "tag:GetResources"
- "tag:TagResources"
Resource: "*"
-
Effect: "Allow"
Action:
- "waf:GetWebACL"
Resource: "*"So, in case someone comes up to the same problem.
The solution is, when creating the rbac roles, to comment out from the rbac-role.yaml (as provided here) the last part which creates the service account.
Since we already created a service account with eksctl and attached to it the aws policy, we can attach to this service account the rbac permissions also. Then this service account can be used normally in the ingress controller pod to do its magic.