Label change of GKE terraform brought down entire cluster

12/20/2019

I made a change to the terraform code for a node pool on GKE where I only changed the labels associated with the node pool, and it tore down the entire cluster (including master) and recreated it. Now every time I try and make a change to the cluster no matter how big or small, terraform wants to tear down and recreate the entire cluster.

We use Terraform Cloud to manage the cluster.

I did something like this in the commit for a few instance groups:

labels = {
-       role = "nodes"
+       "kops.k8s.io/instancegroup" = "nodes"

And these are the fist few lines of the corresponding terraform plan:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.kubernetes.google_container_cluster.master must be replaced
-/+ resource "google_container_cluster" "master" {

The only thing I can think of that could cause this issue was that a few commits earlier I modified the scopes of the instance groups so that they can read and write from Google Storage, but these changes had already been applied to all instance groups in the cluster.

Found Solution:

Described above was a similar problem to this person: https://github.com/gruntwork-io/terraform-google-gke/issues/59

And the solution was to add ignore_changes to our master config like this: https://github.com/gruntwork-io/terraform-google-gke/pull/60/files

-- user6407048
google-kubernetes-engine
terraform-cloud

1 Answer

1/3/2020

I put my edit into this answer:

Described above was a similar problem to this person: https://github.com/gruntwork-io/terraform-google-gke/issues/59

And the solution was to add ignore_changes to our master config like this: https://github.com/gruntwork-io/terraform-google-gke/pull/60/files

-- user6407048
Source: StackOverflow