I deleted a namespace that has a service that is exposed with nginx-ingress with a Let's Encrypt certificate controlled by cert-manager. Deletion of the namespace is hanging with status Terminating.
It is likely a problem with the internal API as explained here. When I run:
kubectl api-resourcesit returns that the certmanager webhook API isn't reachable:
error: unable to retrieve the complete list of server APIs: webhook.certmanager.k8s.io/v1beta1: the server is currently unable to handle the request
When I run kubectl get apiservices v1beta1.webhook.certmanager.k8s.io -o yaml, for checking its status conditions:
...
service:
name: cert-manager-webhook
namespace: nginx-ingress
port: 443
version: v1beta1
versionPriority: 15
status:
conditions:
- lastTransitionTime: "2020-01-21T15:02:23Z"
message: 'failing or missing response from https://10.24.32.6:10250/apis/webhook.certmanager.k8s.io/v1beta1:
bad status from https://10.24.32.6:10250/apis/webhook.certmanager.k8s.io/v1beta1:
404'
reason: FailedDiscoveryCheck
status: "False"
type: AvailableAll nginx-ingress and cert-manager pods are in good health. I have done an update on certmanager in the time that I have deployed and deleted this namespace, which might be an explanation of the issue. How can this problem be solved?
versions:
A simle solution to solve the issue is presented here. But this does not describe how such a problem arises or can be prevented.
Create a temporary JSON file that describes the terminating namespace:
kubectl get namespace <terminating-namespace> -o json >tmp.jsonEdit the file tmp.json by removing the kubernetes value from the finalizers field and save the file.
Set a temporary proxy IP and port:
kubectl proxyFrom a new terminal window, make an API call with your temporary proxy IP and port:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize