K
Q

Question

500 Error while Creating K8S Cluster with Terraform

1/11/2019

Using Terraform, I'm trying to create the most basic possible Kubernetes cluster on GCP, but when I run the config, it retries over and over until it eventually fails with a 500 error.

My main.tf config looks like this:

resource "google_container_cluster" "k8s" {
  name               = "******"
  zone               = "us-east1-c"
  initial_node_count = 3

  master_auth {
    username = "****"
    password = "******"
  }
}

My provider.tf looks like this:

provider "vault" {
  address = "https://*****.domain.com:8200"
  token   = "t0k3n"
}

data "vault_generic_secret" "google" {
  path = "secret/terraform/google"
}

provider "google" {
  project     = "*****"
  region      = "us-east1"
  credentials = "${data.vault_generic_secret.google.data_json}"
}

I store my credential.json file in Vault, and pull it from there. That seems to be working fine - I pull all of my secrets from Vault in other Terraform projects.

Now, my issue is that when I do a terraform apply, it continues to run over and over until it eventually fails with a 500 error. Here's what the debug logs look like:

2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: 2019/01/09 14:35:07 [DEBUG] Google API Request Details:
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: ---[ REQUEST ]---------------------------------------
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: POST /v1beta1/projects/ProjectName/locations/us-east1-c/clusters?alt=json&prettyPrint=false HTTP/1.1
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Host: container.googleapis.com
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: User-Agent: google-api-go-client/0.5 Terraform/0.11.7 (+https://www.terraform.io) terraform-provider-google/1.20.0
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Content-Length: 584
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Content-Type: application/json
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Accept-Encoding: gzip
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: {
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:  "cluster": {
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "binaryAuthorization": {
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "enabled": false
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   },
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "initialNodeCount": 3,
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "legacyAbac": {
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "enabled": false
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   },
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "masterAuth": {
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "password": "****",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "username": "****"
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   },
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "name": "******",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "network": "projects/ProjectName/global/networks/default",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "nodeConfig": {
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "oauthScopes": [
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/devstorage.read_only",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/logging.write",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/monitoring",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/service.management.readonly",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/servicecontrol",
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/trace.append"
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    ]
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   }
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:  }
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: }
2019-01-09T14:35:07.384-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-09T14:35:07.385-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: -----------------------------------------------------
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: 2019/01/09 14:35:07 [DEBUG] Google API Response Details:
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: ---[ RESPONSE ]--------------------------------------
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: HTTP/2.0 500 Internal Server Error
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Cache-Control: private
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Content-Type: application/json; charset=UTF-8
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Date: Wed, 09 Jan 2019 19:35:07 GMT
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Server: ESF
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Vary: Origin
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Vary: X-Origin
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Vary: Referer
2019-01-09T14:35:07.521-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: X-Content-Type-Options: nosniff
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: X-Frame-Options: SAMEORIGIN
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: X-Xss-Protection: 1; mode=block
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: {
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "error": {
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "code": 500,
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "message": "Internal error encountered.",
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "errors": [
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:       {
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:         "message": "Internal error encountered.",
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:         "domain": "global",
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:         "reason": "backendError"
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:       }
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     ],
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "status": "INTERNAL"
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   }
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: }
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: -----------------------------------------------------
2019-01-09T14:35:07.522-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: 2019/01/09 14:35:07 [TRACE] Waiting 1s before next try

The real error looks like this

   -----------------------------------------------------
   2019/01/09 14:35:07 [DEBUG] Google API Response Details:
   ---[ RESPONSE ]--------------------------------------
   HTTP/2.0 500 Internal Server Error
   Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
   Cache-Control: private
   Content-Type: application/json; charset=UTF-8
   Date: Wed, 09 Jan 2019 19:35:07 GMT
   Server: ESF
   Vary: Origin
   Vary: X-Origin
   Vary: Referer
   X-Content-Type-Options: nosniff
   X-Frame-Options: SAMEORIGIN
   X-Xss-Protection: 1; mode=block

   {
   "error": {
   "code": 500,
   "message": "Internal error encountered.",
   "errors": [
   {
   "message": "Internal error encountered.",
   "domain": "global",
   "reason": "backendError"
   }
   ],
   "status": "INTERNAL"
   }
   }

This config can't really get any simpler, yet I'm completely unable to deploy it. I've made sure that the appropriate API is enabled and I've given it some time in case it wasn't fully "ready", but it continues to fail with the same 500 "internal error occurred" message.

For Terraform, I'm using a service account that has been given Project Owner IAM permissions. I'm also able to create clusters manually in the GUI.

Any thoughts on how to get past this?

-- CoreyF

google-cloud-platform

google-kubernetes-engine

kubernetes

terraform

K
Q

500 Error while Creating K8S Cluster with Terraform

Similar Questions

0 Answers

KQ

500 Error while Creating K8S Cluster with Terraform

Similar Questions

0 Answers

K
Q