Using an APIService resource I registered an add-on API server with my core API Server. Now, for whatever reason, this add-on API server has become unresponsive. The problem is that kubectl works by getting information about all the versions of all the groups it can see when running kubectl get --raw "/apis", including my unresponsive APIService and so it is hanging whenever I type any command that contacts the API Server and I can not longer administer my cluster.
Is there a good way to deal with this situation?
All Kubernetes objects' data is stored in etcd backend. Just delete unresponsive APIService from there.
You can find it in etcd version 3 the following way:
$ ETCDCTL_API=3 etcdctl --endpoints=<etcd_ip>:2379 get / --prefix --keys-only | grep -i apiservice
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.crd.projectcalico.org
/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.storage.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1alpha1.monitoring.coreos.com
/registry/apiregistration.k8s.io/apiservices/v1alpha1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1alpha1.settings.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.apps
/registry/apiregistration.k8s.io/apiservices/v1beta1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.certificates.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.extensions
/registry/apiregistration.k8s.io/apiservices/v1beta1.policy
/registry/apiregistration.k8s.io/apiservices/v1beta1.rbac.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.storage.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1beta1.voyager.appscode.com
/registry/apiregistration.k8s.io/apiservices/v2alpha1.batchAnd delete it after that:
ETCDCTL_API=3 etcdctl --endpoints=10.128.10.11:2379 del <path>