I'm bring a GKE cluster up with Terraform, which works nicely. I then want Terraform to perform some Kubernetes-level operations on the cluster (using the k8s provider) - nothing massive, just installing a couple of Deployments etc.
The problem I'm having is permissions. I'd like a neat, tidy, declarative way to make a cluster and have a set of credentials in hand that I can use short-term to do "admin" operations on it, including bootstrapping other users. I know how to make the google user that's running TF an admin of the cluster (that question comes up a lot), but that doesn't seem very nice. Not least, the k8s TF provider doesn't support clusterolebinding (Issue, partial PR) so you have to "shell out" with a local-exec provisioner to first run gcloud container clusters get-credentials and then kubectl create clusterrolebinding ....
Similarly, I don't want to set a master password, because I don't want to run with HTTP basic auth on. The nicest option looks to be the key/cert pair that's returned by the TF GKE resource, but that has a CN of "client", and that user has no power. So again, the only way to use it is to shell out to kubectl, pass it the gcloud service account credentials, and get it to add a clusterrolebinding for "client", at which point I may as well just do everything as the service account like above.
For contrast, on EKS the (AWS IAM) user that creates the cluster has cluster-admin out of the box (I assume the AWS authn provider claim's the user is in "system:masters").
My actual question here is: is there a neat, fully declarative way in Terraform to bring up a cluster and have available (ideally as output) a potent set of credentials to use and then drop? (yes I know they'll stay in the tfstate)
My options seem to be:
And I don't like any of them!
I've been running into a similar problem, which has gotten particularly nasty since a recent Kubernetes issue unexpectedly disabled basic auth by default, which broke my previously-functioning Terraform configuration as soon as I tried to build a new cluster from the same config.
Finally found an answer in this SO answer, which recommends a method of using Terraform's Google IAM creds to connect to the cluster without needing the "shell out". Note that this method allows cluster permissions to be bootstrapped in Terraform with no external tooling/hacks/etc and without needing to have basic auth enabled.
The relevant part of that answer is:
data "google_client_config" "default" {}
provider "kubernetes" {
host = "${google_container_cluster.default.endpoint}"
token = "${data.google_client_config.default.access_token}"
cluster_ca_certificate = "${base64decode(google_container_cluster.default.master_auth.0.cluster_ca_certificate)}"
load_config_file = false
}