K
Q

Cert-manager fails on kubernetes with webhooks

6/25/2019

I'm following the Kubernetes install instructions for Helm: https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html With Cert-manager v0.81 on K8 v1.15, Ubuntu 18.04 on-premise. When I get to testing the installation, I get these errors:

error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "issuers.admission.certmanager.k8s.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "certificates.admission.certmanager.k8s.io": the server is currently unable to handle the request

If I apply the test-resources.yaml before installing with Helm, I'm not getting the errors but it is still not working. These errors are new to me, as Cert-manager used to work for me on my previous install about a month ago, following the same installation instructions. I've tried with Cert-Manager 0.72(CRD 0.7) as well as I think that was the last version I managed to get installed but its not working either.

What does these errors mean?

Update: It turned out to be an internal CoreDNS issue on my cluster. Somehow not being configured correctly. Possible related to wrong POD_CIDR configuration.

-- Isaack Rasmussen
cert-manager
kubernetes
kubernetes-helm

Similar Questions

Jetstack Cert not able to challenge
What is the "minikube docker-env" replacement for Docker for Windows
Docker add Javafx to openJDK
Is it possible for stackdriver to recognise syslog input from k8s?
What is the difference between patches vs patchesJson6902 in Kustomize

2 Answers

6/28/2019

I also tried cert-manager v0.8.0 a very similar setup on Ubuntu 18.04 and k8s v1.14.1 and I began to get the same error when i tore down cert-manager using kubectl delete and reinstalled it, after experiencing some network issues on the cluster.

I stumbled on a solution that worked. On the master node, simply restart the apiserver container:

$ sudo docker ps -a | grep apiserver
af99f816c7ec        gcr.io/google_containers/kube-apiserver@sha256:53b987e5a2932bdaff88497081b488e3b56af5b6a14891895b08703129477d85               "/bin/sh -c '/usr/loc"   15 months ago       Up 19 hours                                     k8s_kube-apiserver_kube-apiserver-ip-xxxxxc_0
40f3a18050c3        gcr.io/google_containers/pause-amd64:3.0                                                                                      "/pause"                 15 months ago       Up 15 months                                    k8s_POD_kube-apiserver-ip-xxxc_0
$ sudo docker restart af99f816c7ec
af99f816c7ec
$

Then try applying the test-resources.yaml again:

$ kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
issuer.certmanager.k8s.io/test-selfsigned created
certificate.certmanager.k8s.io/selfsigned-cert created

If that does not work, this github issue mentions that the master node might need firewall rules to be able to reach the cert-manager-webhook pod. The exact steps to do so will depend on which cloud platform you are on.

-- L. J.
Source: StackOverflow
7/7/2019

If you experience this problem, check the logs of CoreDNS(Or KubeDNS) and you may see lots of errors related to contacting services. Unfortunately, I no longer have the errors. But this is how I figured out that my network setup was invalid.

I'm using Calico(Will apply for other networks as well) and its network was not set to the same as the POD_CIDR network that I initialized my Kubernetes with.

Example 1. Set up K8:

kubeadm init --pod-network-cidr=10.244.0.0/16

  1. Configure Calico.yaml:

    - name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
-- Isaack Rasmussen
Source: StackOverflow