K
Q

Change Public GKE to Private GKE cluster using terraform

1/27/2021

How to change the existing GKE cluster to GKE private cluster? Will I be able to connect to the Kubectl API from internet based on firewall rules or should I have a bastion host? I don't want to implement Cloud Nat or nat gateway. I have a squid proxy VM that can handle internet access for pods. I just need to be able to connect to Kubectl to apply or modify anything.

I'm unsure how to modify the existing module I wrote to make the nodes private and I'm not sure if the cluster will get deleted if I try and apply the new changes related to private gke cluster.

resource "google_container_cluster" "primary" {
  name     = "prod"
  network  = "prod"
  subnetwork = "private-subnet-a"
  location               = "us-west1-a"
  remove_default_node_pool = true
  initial_node_count = 1

  depends_on = [var.depends_on_vpc]
}

resource "google_container_node_pool" "primary_nodes" {
  depends_on = [var.depends_on_vpc]

  name       = "prod-node-pool"
  location   = "us-west1-a"
  cluster    = google_container_cluster.primary.name
  node_count = 2

  node_config {
    preemptible  = false
    machine_type = "n1-standard-2"

    metadata = {
      disable-legacy-endpoints = "true"
    }

    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/compute",
    ]
  }
}
-- John Doe
google-kubernetes-engine
kubernetes
terraform

Similar Questions

Cloud container clusters create `compute.networks.get` permission error
Ingress: Connection refused, however, it works from cluster
GKE - Unable to make cuda work with pytorch
Goroutine opening a new connection to database after each request (sqlx) and ticker
AWS has per node Pod IP restrictions, pods are stuck at ContainerCreating state

2 Answers

4/10/2021

1) You will have to recreate the cluster since the private/public option is immutable. Terraform will recreate the cluster. 2) To access the private cluster endpoints, you can choose appropriate methods

   1) Public endpoint access disabled:- creates a private cluster with no client access to the public endpoint.
   2) Public endpoint access enabled, authorized networks enabled:- creates a private cluster with limited access to the public endpoint.
   3) Public endpoint access enabled, authorized networks disabled:- creates a private cluster with unrestricted access to the public endpoint.

3) To ssh into the node/pod from the authorized network, you can setup access via IAP.

I am using this terraform module to manage multiple clusters with the 2nd option, it is fully configurable.

-- Jijo John
Source: StackOverflow
1/27/2021

Answering the part of the question:

How to change the existing GKE cluster to GKE private cluster?

GKE Private cluster network settings

GKE setting: Private cluster is immutable. This setting can only be set during the GKE cluster provisioning.

To create your cluster as a private one you can either:

  • Create a new GKE private cluster.
  • Duplicate existing cluster and set it to private:
    • This setting is available in GCP Cloud Console -> Kubernetes Engine -> CLUSTER-NAME -> Duplicate
    • This setting will clone the configuration of your infrastructure of your previous cluster but not the workload (Pods, Deployments, etc.)

Will I be able to connect to the Kubectl API from internet based on firewall rules or should I have a bastion host?

Yes, you could but it will heavily depend on the configuration that you've chosen during the GKE cluster creation process.

As for ability to connect to your GKE private cluster, there is a dedicated documentation about it:

As for how you can create a private cluster with Terraform, there is the dedicated site with configuration options specific to GKE. There are also parameters responsible for provisioning a private cluster:

As for a basic example of creating a private GKE cluster with Terraform:

  • main.tf
provider "google" {
  project = "INSERT_PROJECT_HERE" 
  region  = "europe-west3"
  zone    = "europe-west3-c"
}
  • gke.tf
resource "google_container_cluster" "primary-cluster" {
  name               = "gke-private"
  location           = "europe-west3-c"
  initial_node_count = 1

  private_cluster_config {
    enable_private_nodes = "true"
    enable_private_endpoint = "false" # this option will make your cluster available through public endpoint 
    master_ipv4_cidr_block = "172.16.0.0/28"
  }

  ip_allocation_policy {
    cluster_secondary_range_name = "" 
    services_secondary_range_name = ""
  }

  
  node_config {
    machine_type = "e2-medium"
  }
}

A side note!

I've created a public GKE cluster, modified the .tf responsible for it's creation to support private cluster. After running: $ terraform plan Terraform responded with the information that the cluster will be recreated.

-- Dawid Kruk
Source: StackOverflow