We've set-up a new ingress route that requires TLS certificate authentication, and we have placed it on its own subdomain, but we are finding that cert-manager is unable to issue a certificate for it.
Using the examples provided here, we generated the CA cert and CA key, and then configured the client certificate: https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/auth/client-certs
In the logs, I can see that all of the .acme-challenge requests are returning a 403. I am guessing that nginx is rejecting the requests because Let's Encrypt can't present a client certificate for the challenge request. What do I need to do in order to bypass the client cert requirement for ACME?
The issue here was that we actually had misconfigured the nginx.ingress.kubernetes.io/auth-tls-secret annotation. It must be in namespace/name format -- where namespace is the namespace that contains the secret containing the client CA certificate, and name is the name of that secret -- but we were only providing the name since the secret is in the same namespace with the ingress.
I was able to diagnose the issue by dumping the nginx ingress controller config to nginx.conf.txt with:
kubectl exec <NAME OF INGRESS CONTROLLER POD> -n <INGRESS NAMESPACE> -- nginx -T | tee nginx.conf.txt(Adapted from https://docs.nginx.com/nginx-ingress-controller/troubleshooting/#checking-the-generated-config).
This included the following snippet:
## start server the.hostname.com
server {
server_name the.hostname.com ;
listen 80;
set $proxy_upstream_name "-";
set $pass_access_scheme $scheme;
set $pass_server_port $server_port;
set $best_http_host $http_host;
set $pass_port $pass_server_port;
listen 443 ssl http2;
# PEM sha: 66c07c44ba9637a23cd3d7b6ebce958e08a52ccb
ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem;
ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem;
ssl_certificate_by_lua_block {
certificate.call()
}
# Location denied, reason: invalid format (namespace/name) found in 'the-secret-name'
return 403;
}
## end server the.hostname.comThe key is these two lines:
# Location denied, reason: invalid format (namespace/name) found in 'the-secret-name'
return 403;This pointed me to the annotation for the secret name. Once I fixed that, ACME worked properly.