I'm trying to get letsencrypt/cert-manager running via this Helm chart. The K8s cluster is on Digital Ocean.
I successfully verified the installation as recommended and have created a ClusterIssuer
for staging, and 1 for production. (letsencrypt-staging
, letsencrypt-prod
)
Problem: The acme challenge returns a 404 error.
$ k get challenge -o wide
NAME STATE DOMAIN REASON AGE
myapp-cert-2315925673-2905389610-1118496475 pending myapp.example.com Waiting for http-01 challenge propagation: wrong status code '404', expected '200' 7m55s
The Ingress works fine with port 80 when the tls
block commented out. When I define tls
however, requests on port 80 return a 404, which is probably why the challenge is failing.
Note: I get the same response when using my production ClusterIssuer
.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myapp-ingress
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-staging
labels:
app: myapp
spec:
rules:
- host: myapp.example.com
http:
paths:
- backend:
serviceName: myapp
servicePort: 80
tls:
- hosts:
- myapp.example.com
secretName: myapp-cert
:: edited to add more configs ::
After adding more configs and logs as requested by @Tubc, it appears that Nginx is throwing an error when I update the ingress because the cert doesn't exist.
ClusterIssuer Manifests:
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: me@example.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: me@example.com
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: nginx
Service Manifest:
---
apiVersion: v1
kind: Service
metadata:
name: myapp
labels:
app: myapp
spec:
ports:
- port: 80
selector:
app: myapp
tier: fe
type: NodePort
Nginx log:
2019/12/08 14:45:44 [emerg] 62#62: cannot load certificate "/etc/nginx/secrets/default-myapp-cert": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE) I1208 14:45:44.934644 1 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"myapp-ingress", UID:"610c3304-0565-415d-8cde-0863bf9325ca", APIVersion:"extensions/v1beta1", ResourceVersion:"319124", FieldPath:""}): type: 'Warning' reason: 'AddedOrUpdatedWithError' Configuration for default/myapp-ingress was added or updated, but not applied: Error reloading NGINX for default/myapp-ingress: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: "" stderr: "nginx: [emerg] cannot load certificate \"/etc/nginx/secrets/default-myapp-cert\": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)\n" finished with error: exit status 1
Despite copying the docs, it turns out the annotation key on the Ingress was wrong.
It should be: certmanager.k8s.io/cluster-issuer
(rather than cert-manager.io/cluster-issuer
as documented)
Upon making this change the 404 went away and the certificate was issued and configured properly.