I have launched a private GKE cluster using terraform resource "google_container_cluster" with a private_cluster_config block in it.

I have added master_authorized_networks_config to allow my own IP address in authorized networks for the GKE.

And I have added k8s namespace using terraform resource "kubernetes_namespace".

I have also set all google, kubernetes providers, k8s token, cluster_ca_certificate etc correctly and the namespace was indeed provisioned by this terraform.

resource "google_container_cluster" "k8s_cluster" {
  # .....
  # .....
  private_cluster_config {
    enable_private_nodes = true
    enable_private_endpoint = false
    master_ipv4_cidr_block = "172.16.0.0/28"
  }
  ip_allocation_policy { } # enables VPC-native
  master_authorized_networks_config {
    cidr_blocks {
      {
        cidr_block = "0.0.0.0/0"
        display_name = "World"
      }
    }
  }
  # .....
  # .....
}

data "google_client_config" "google_client" {}

data "google_container_cluster" "k8s_cluster" {
  name     = google_container_cluster.k8s_cluster.name
  location = var.location
}

provider "kubernetes" {
  # following this example https://www.terraform.io/docs/providers/google/d/datasource_client_config.html#example-usage-configure-kubernetes-provider-with-oauth2-access-token
  version = "1.11.1"
  load_config_file = false
  host = google_container_cluster.k8s_cluster.endpoint
  token = data.google_client_config.google_client.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.k8s_cluster.master_auth.0.cluster_ca_certificate
  )
}

resource "kubernetes_namespace" "namespaces" {
  depends_on = [google_container_node_pool.node_pool]
  for_each = ["my-ns"]
  metadata {
    name = each.value
  }
}

Then I ran terraform apply and the namespace was created fine ✅✅✅

kubernetes_namespace.namespaces["my-ns"]: Creating...
kubernetes_namespace.namespaces["my-ns"]: Creation complete after 1s [id=my-ns]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

However, when I run terraform apply or terraform plan again and terraform is trying to refresh the namespace resource,

data.google_container_cluster.k8s_cluster: Refreshing state...
kubernetes_namespace.namespaces["my-ns"]: Refreshing state... [id=my-ns]

it's throwing the following error intermittently. ❌ ❌ ❌

Error: Get http://localhost/api/v1/namespaces/my-ns: dial tcp 127.0.0.1:80: connect: connection refused

It's sometimes passing and sometimes failing - intermittently.

Where would you advise I should look into to fix this intermittent error?