After deleting kubernetes cluster with "terraform destroy" I can't create it again anymore.
"terraform apply" returns the following error message:
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Here is the terraform configuration:
terraform {
backend "s3" {
bucket = "skyglass-msur"
key = "terraform/backend"
region = "us-east-1"
}
}
locals {
env_name = "staging"
aws_region = "us-east-1"
k8s_cluster_name = "ms-cluster"
}
variable "mysql_password" {
type = string
description = "Expected to be retrieved from environment variable TF_VAR_mysql_password"
}
provider "aws" {
region = local.aws_region
}
data "aws_eks_cluster" "msur" {
name = module.aws-kubernetes-cluster.eks_cluster_id
}
module "aws-network" {
source = "github.com/skyglass-microservices/module-aws-network"
env_name = local.env_name
vpc_name = "msur-VPC"
cluster_name = local.k8s_cluster_name
aws_region = local.aws_region
main_vpc_cidr = "10.10.0.0/16"
public_subnet_a_cidr = "10.10.0.0/18"
public_subnet_b_cidr = "10.10.64.0/18"
private_subnet_a_cidr = "10.10.128.0/18"
private_subnet_b_cidr = "10.10.192.0/18"
}
module "aws-kubernetes-cluster" {
source = "github.com/skyglass-microservices/module-aws-kubernetes"
ms_namespace = "microservices"
env_name = local.env_name
aws_region = local.aws_region
cluster_name = local.k8s_cluster_name
vpc_id = module.aws-network.vpc_id
cluster_subnet_ids = module.aws-network.subnet_ids
nodegroup_subnet_ids = module.aws-network.private_subnet_ids
nodegroup_disk_size = "20"
nodegroup_instance_types = ["t3.medium"]
nodegroup_desired_size = 1
nodegroup_min_size = 1
nodegroup_max_size = 5
}
# Create namespace
# Use kubernetes provider to work with the kubernetes cluster API
provider "kubernetes" {
# load_config_file = false
cluster_ca_certificate = base64decode(data.aws_eks_cluster.msur.certificate_authority.0.data)
host = data.aws_eks_cluster.msur.endpoint
exec {
api_version = "client.authentication.k8s.io/v1alpha1"
command = "aws-iam-authenticator"
args = ["token", "-i", "${data.aws_eks_cluster.msur.name}"]
}
}
# Create a namespace for microservice pods
resource "kubernetes_namespace" "ms-namespace" {
metadata {
name = "microservices"
}
}
P.S. There seems to be the issue with terraform kubernetes provider for 0.14.7
I couldn't use "load_config_file" = false in this version, so I had to comment it, which seems to be the reason of this issue.
P.P.S. It could also be the issue with outdated cluster_ca_certificate, which terraform tries to use: deleting this certificate could be enough, although I'm not sure, where it is stored.
Deleting .terraform sub-folder in the folder where you run "terraform" command should also solve the issue.
I didn't try it for this exact situation, but I had a similar issue today, so I decided to share another solution. It seems less radical, than deleting S3 bucket.
Before doing something radical like manipulating the state directly, try setting the KUBE_CONFIG_PATH variable:
export KUBE_CONFIG_PATH=/path/to/.kube/config
After this rerun the plan
or apply
command.
This has fixed the issue for me.
I had the same issue. I even manually deleted the EKS cluster which really messed up the terraform state.
However, after wasting a few hours, I found out that there is a very simple solution.
You can run
terraform state rm <resource_type>.<resource_name>
I just executed
terraform state rm `terraform state list | grep eks`
to remove all the entries for a particular service from state file in a safe manner.
Deleting terraform state S3 bucket on AWS solved the issue.