AWS supports IAM Roles for Service Accounts (IRSA) that allows cluster operators to map AWS IAM Roles to Kubernetes Service Accounts.
To do so, one has to create an iamserviceaccount in an EKS cluster:
eksctl create iamserviceaccount \
--name <AUTOSCALER_NAME> \
--namespace kube-system \
--cluster <CLUSTER_NAME> \
--attach-policy-arn <POLICY_ARN> \
--approve \
--override-existing-serviceaccountsThe problem is that I don't want to use the above eksctl command because I want to declare my infrastructure using terraform.
Does eksctl command do anything other than creating a service account? If it only creates a service account, what is the YAML representation of it?
A role created for that purpose looks like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region>.amazonaws.com/id/<oidc-id>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.<region>.amazonaws.com/id/<oidc-id>": "system:serviceaccount:<kube-serviceaccount-namespace>:<kube-serviceaccount-name>"
}
}
}
]
}
I highly recommend you to use iam_assumable_role_admin terraform module for creating this IAM Role for you.
I am adding my answer here because I stumble upon the same issue, and accepted answer (and other answers above), do not provide full resolution to the issue - no code examples. They are just guidelines which I had to use to research much deeper. There are some issues which is really easy to miss - and without code examples its quite hard to conclude what is happening (especially part related with Conditions/StringEquals while creating IAM role)
The whole purpose of creating a service account which is going to be tied with the role - is possibility of creating aws resources from within cluster (most common case is load balancer, or roles for pushing logs to the cloudwatch).
So, question is how we can do this, using terraform, instead of using eks commands.
What we need to do, is: 1. create eks oidc (which can be done with terraform) 2. create AWS IAM role (which can be done with terraform), create and use proper policies 3. Create k8s service account (needs to be done with kubectl commands - or with terraform using kubernetes resources 4. Annotate k8s service account with IAM role we created (meaning that we are linking k8s service account with IAM role)
After this setup, our k8s service account will have k8s cluster role and k8s cluster role binding (which will allow that service account to perform actions within the k8s) and, our k8s service account will have IAM role attached to it, which will allow to perform actions outside of the cluster (like creating aws resources)
So lets start with it. Assumption bellow is that your eks cluster is already created with terraform, and we are focusing on creating resources areound that eks cluster necessary for working service account.
Create eks_oidc
### First we need to create tls certificate
data "tls_certificate" "eks-cluster-tls-certificate" {
url = aws_eks_cluster.eks-cluster.identity[0].oidc[0].issuer
}
# After that create oidc
resource "aws_iam_openid_connect_provider" "eks-cluster-oidc" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.eks-cluster-tls-certificate.certificates[0].sha1_fingerprint]
url = aws_eks_cluster.eks-cluster.identity[0].oidc[0].issuer
}
Now, lets create AWS IAM role with all necessary policies.
Terraform declarative code bellow will:
Make a note that i used suffixes as alb ingress controller here, because that is primary use of my role from within the cluster. You can change the name of policy of the role or you can change permission access for the policy as well in dependency of what you are planing to do with it.
data "aws_caller_identity" "current" {}
locals {
account_id = data.aws_caller_identity.current.account_id
eks_oidc = replace(replace(aws_eks_cluster.eks-cluster.endpoint, "https://", ""), "/\\..*$/", "")
}
# Policy which will allow us to create application load balancer from inside of cluster
resource "aws_iam_policy" "ALBIngressControllerIAMPolicy" {
name = "ALBIngressControllerIAMPolicy"
description = "Policy which will be used by role for service - for creating alb from within cluster by issuing declarative kube commands"
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Effect = "Allow",
Action = [
"elasticloadbalancing:ModifyListener",
"wafv2:AssociateWebACL",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:DescribeInstances",
"wafv2:GetWebACLForResource",
"elasticloadbalancing:RegisterTargets",
"iam:ListServerCertificates",
"wafv2:GetWebACL",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:SetWebAcl",
"ec2:DescribeInternetGateways",
"elasticloadbalancing:DescribeLoadBalancers",
"waf-regional:GetWebACLForResource",
"acm:GetCertificate",
"shield:DescribeSubscription",
"waf-regional:GetWebACL",
"elasticloadbalancing:CreateRule",
"ec2:DescribeAccountAttributes",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"waf:GetWebACL",
"iam:GetServerCertificate",
"wafv2:DisassociateWebACL",
"shield:GetSubscriptionState",
"ec2:CreateTags",
"elasticloadbalancing:CreateTargetGroup",
"ec2:ModifyNetworkInterfaceAttribute",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"ec2:RevokeSecurityGroupIngress",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"shield:CreateProtection",
"acm:DescribeCertificate",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:DescribeRules",
"ec2:DescribeSubnets",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"waf-regional:AssociateWebACL",
"tag:GetResources",
"ec2:DescribeAddresses",
"ec2:DeleteTags",
"shield:DescribeProtection",
"shield:DeleteProtection",
"elasticloadbalancing:RemoveListenerCertificates",
"tag:TagResources",
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:DescribeListeners",
"ec2:DescribeNetworkInterfaces",
"ec2:CreateSecurityGroup",
"acm:ListCertificates",
"elasticloadbalancing:DescribeListenerCertificates",
"ec2:ModifyInstanceAttribute",
"elasticloadbalancing:DeleteRule",
"cognito-idp:DescribeUserPoolClient",
"ec2:DescribeInstanceStatus",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:CreateLoadBalancer",
"waf-regional:DisassociateWebACL",
"elasticloadbalancing:DescribeTags",
"ec2:DescribeTags",
"elasticloadbalancing:*",
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:DeleteTargetGroup",
"ec2:DescribeSecurityGroups",
"iam:CreateServiceLinkedRole",
"ec2:DescribeVpcs",
"ec2:DeleteSecurityGroup",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:DescribeTargetGroups",
"shield:ListProtections",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:DeleteListener"
],
Resource = "*"
}
]
})
}
# Create IAM role
resource "aws_iam_role" "alb-ingress-controller-role" {
name = "alb-ingress-controller"
assume_role_policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "${aws_iam_openid_connect_provider.eks-cluster-oidc.arn}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${replace(aws_iam_openid_connect_provider.eks-cluster-oidc.url, "https://", "")}:sub": "system:serviceaccount:kube-system:alb-ingress-controller",
"${replace(aws_iam_openid_connect_provider.eks-cluster-oidc.url, "https://", "")}:aud": "sts.amazonaws.com"
}
}
}
]
}
POLICY
depends_on = [aws_iam_openid_connect_provider.eks-cluster-oidc]
tags = {
"ServiceAccountName" = "alb-ingress-controller"
"ServiceAccountNameSpace" = "kube-system"
}
}
# Attach policies to IAM role
resource "aws_iam_role_policy_attachment" "alb-ingress-controller-role-ALBIngressControllerIAMPolicy" {
policy_arn = aws_iam_policy.ALBIngressControllerIAMPolicy.arn
role = aws_iam_role.alb-ingress-controller-role.name
depends_on = [aws_iam_role.alb-ingress-controller-role]
}
resource "aws_iam_role_policy_attachment" "alb-ingress-controller-role-AmazonEKS_CNI_Policy" {
role = aws_iam_role.alb-ingress-controller-role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
depends_on = [aws_iam_role.alb-ingress-controller-role]
}
After executing terraform above, you have successfully created terraform part of the resources. Now we need to create a k8s service account and bind IAM role with that service account.
Creating cluster role, cluster role binding and service account
You can use
directly (from the master branch), but having in mind that we need to annotate the iam arn, i have tendency to download this file, update it and store it as updated within my kubectl config files.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
rules:
- apiGroups:
- ""
- extensions
resources:
- configmaps
- endpoints
- events
- ingresses
- ingresses/status
- services
- pods/status
verbs:
- create
- get
- list
- update
- watch
- patch
- apiGroups:
- ""
- extensions
resources:
- nodes
- pods
- secrets
- services
- namespaces
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: alb-ingress-controller
subjects:
- kind: ServiceAccount
name: alb-ingress-controller
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/name: alb-ingress-controller
name: alb-ingress-controller
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: <ARN OF YOUR ROLE HERE>
...At the bottom of this file, you will notice annotation where you will need to place your ANR role.
Double check
And that would be it. After that you have a k8s service account which is connected with iam role.
Check with:
kubectl get sa -n kube-system
kubectl describe sa alb-ingress-controller -n kube-systemAnd you should get output similar to this (annotations is the most important part, because it confirms the attachment of iam role):
Name: alb-ingress-controller
Namespace: kube-system
Labels: app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=alb-ingress-controller
Annotations: eks.amazonaws.com/role-arn: <YOUR ANR WILL BE HERE>
meta.helm.sh/release-name: testrelease
meta.helm.sh/release-namespace: default
Image pull secrets: <none>
Mountable secrets: alb-ingress-controller-token-l4pd8
Tokens: alb-ingress-controller-token-l4pd8
Events: <none>From now on, you can use this service to manage internal k8s resources and external which are allowed by the policies you attached.
In my case, as mentioned before, I used it (beside other things) for creation of alb ingress controller and load balancer, hence all of the prefixes with "alb-ingress"
First, you should define IAM role in Terraform.
Second, you should configure aws-auth configmap in Kubernetes to map the IAM role to Kubernetes user or serviceaccount. You can do that in Terraform using Kubernetes provider.
There is already a Terraform module terraform-aws-eks which manages all aspects of EKS cluster. You may take some ideas from it.
After Vasili Angapov's helps, now I can answer the question:
Yes It does more than just creating a service account. It does three things:
Now It's easy to declare the above steps using kubernetes and aws providers in terraform.