My Terraform code describes some AWS infrastructure to build a Kubernetes cluster including some deployments into the cluster. When I try to destroy the infrastructure using terraform plan -destroy
I get a cycle:
module.eks_control_plane.aws_eks_cluster.this[0] (destroy)
module.eks_control_plane.output.cluster
provider.kubernetes
module.aws_auth.kubernetes_config_map.this[0] (destroy)
data.aws_eks_cluster_auth.this[0] (destroy)
Destroying the infrastructure works by hand using just terraform destroy
works fine. Unfortunately, Terraform Cloud uses terraform plan -destroy
to plan the destructuion first, which causes this to fail. Here is the relevant code:
excerpt from eks_control_plane module:
resource "aws_eks_cluster" "this" {
count = var.enabled ? 1 : 0
name = var.cluster_name
role_arn = aws_iam_role.control_plane[0].arn
version = var.k8s_version
# https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
enabled_cluster_log_types = var.control_plane_log_enabled ? var.control_plane_log_types : []
vpc_config {
security_group_ids = [aws_security_group.control_plane[0].id]
subnet_ids = [for subnet in var.control_plane_subnets : subnet.id]
}
tags = merge(var.tags,
{
}
)
depends_on = [
var.dependencies,
aws_security_group.node,
aws_iam_role_policy_attachment.control_plane_cluster_policy,
aws_iam_role_policy_attachment.control_plane_service_policy,
aws_iam_role_policy.eks_cluster_ingress_loadbalancer_creation,
]
}
output "cluster" {
value = length(aws_eks_cluster.this) > 0 ? aws_eks_cluster.this[0] : null
}
aws-auth Kubernetes config map from aws_auth module:
resource "kubernetes_config_map" "this" {
count = var.enabled ? 1 : 0
metadata {
name = "aws-auth"
namespace = "kube-system"
}
data = {
mapRoles = jsonencode(
concat(
[
{
rolearn = var.node_iam_role.arn
username = "system:node:{{EC2PrivateDNSName}}"
groups = [
"system:bootstrappers",
"system:nodes",
]
}
],
var.map_roles
)
)
}
depends_on = [
var.dependencies,
]
}
Kubernetes provider from root module:
data "aws_eks_cluster_auth" "this" {
count = module.eks_control_plane.cluster != null ? 1 : 0
name = module.eks_control_plane.cluster.name
}
provider "kubernetes" {
version = "~> 1.10"
load_config_file = false
host = module.eks_control_plane.cluster != null ? module.eks_control_plane.cluster.endpoint : null
cluster_ca_certificate = module.eks_control_plane.cluster != null ? base64decode(module.eks_control_plane.cluster.certificate_authority[0].data) : null
token = length(data.aws_eks_cluster_auth.this) > 0 ? data.aws_eks_cluster_auth.this[0].token : null
}
And this is how the modules are called:
module "eks_control_plane" {
source = "app.terraform.io/SDA-SE/eks-control-plane/aws"
version = "0.0.1"
enabled = local.k8s_enabled
cluster_name = var.name
control_plane_subnets = module.vpc.private_subnets
k8s_version = var.k8s_version
node_subnets = module.vpc.private_subnets
tags = var.tags
vpc = module.vpc.vpc
dependencies = concat(var.dependencies, [
# Ensure that VPC including all security group rules, network ACL rules,
# routing table entries, etc. is fully created
module.vpc,
])
}
# aws-auth config map module. Creating this config map will allow nodes and
# Other users to join the cluster.
# CNI and CSI plugins must be set up before creating this config map.
# Enable or disable this via `aws_auth_enabled` variable.
# TODO: Add Developer and other roles.
module "aws_auth" {
source = "app.terraform.io/SDA-SE/aws-auth/kubernetes"
version = "0.0.0"
enabled = local.aws_auth_enabled
node_iam_role = module.eks_control_plane.node_iam_role
map_roles = [
{
rolearn = "arn:aws:iam::${var.aws_account_id}:role/Administrator"
username = "admin"
groups = [
"system:masters",
]
},
{
rolearn = "arn:aws:iam::${var.aws_account_id}:role/Terraform"
username = "terraform"
groups = [
"system:masters",
]
}
]
}
Removing the aws_auth config map, which means not using the Kubernetes provider at all, breaks the cycle. The problem is obviously that Terraform tries to destroys the Kubernetes cluster, which is required for the Kubernetes provider. Manually removing the resources step by step using multiple terraform apply
steps works fine, too.
Is there a way that I can tell Terraform first to destroy all Kubernetes resources so that the Provider is not required anymore, then destroy the EKS cluster?