I've been running an app on Kubernetes very smoothly for the past few weeks, until today I found that my Kubernetes master is no longer available.
But the application is still up, seemingly because the cluster's nodes are still available in Google Compute Engine:
(We don't use any compute instances aside from our K8s cluster.)
My gcloud
CLI has all the right configuration:
$ gcloud config list
Your active configuration is: [default]
[compute]
...
[container]
use_client_certificate = True
[core]
account = {email}
disable_usage_reporting = True
project = {myproject}
Even my kubectl
CLI is configured to use the cluster that once existed:
$ kubectl config current-context
gke_{myproject}_us-central1-a_{clustername}
But any kubectl
call results in a timeout.
$ kubectl get pods
Unable to connect to the server: dial tcp {ipv4 address}:443: i/o timeout
gcloud
and the Google Container Engine web UI both agree that there are no clusters:
$ gcloud container clusters list
*crickets*
I got in touch with Google Support to solve this problem. The cluster's master instance was erroneously marked as abusive, and the master was taken down. Google was able to bring it back up. After that, there was a process to return the cluster to its normal state.
I suppose the solution here is to contact support.