cert-manager challenges are pending

7/23/2021

I'm using the cert-manager to manage my ssl certificates in my Kubernetes cluster. The cert-manager creates the pods and the challenges, but the challenges are never getting fulfilled. They're always saying:

Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://somedomain/.well-known/acme-challenge/VqlmMCsb019CCFDggs03RyBLZJ0jo53LO...': Get "http://somedomain/.well-known/acme-challenge/VqlmMCsb019CCFDggs03RyBLZJ0jo53LO...": EOF

But when I open the url (http://<somedomain>/.well-known/acme-challenge/VqlmMCsb019CCFDggs03RyBLZJ0jo53LO...), it returns the expected code:

vzCVdTk1q55MQCNH...zVkKYGvBJkRTvDBHQ.YfUcSfIKvWo_MIULP9jvYcgtsGxwfJMLWUGsB5kFKRc

When I do kubectl get certs, it says that the certs are ready:

NAMEREADYSECRETAGE
crt1Truecrt1-secret65m
crt1-secretTruecrt1-secret65m
crt2Truecrt2-secret65m
crt2-secretTruecrt2-secret65m

It looks like Let's Encrypt never calls (or the cert-manager never instructs) these url's to verify.

When I list the challenges kubectl describe challenges, it says:

Name:         crt-secret-hcgcf-269956107-974455061
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2021-07-23T10:47:27Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"finalizer.acme.cert-manager.io":
        f:ownerReferences:
          .:
          k:{"uid":"09e39ad0-cc39-421f-80d2-07c2f82680af"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:authorizationURL:
        f:dnsName:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:key:
        f:solver:
          .:
          f:http01:
            .:
            f:ingress:
              .:
              f:class:
              f:ingressTemplate:
    UID:                   09e39ad0-cc39-421f-80d2-07c2f82680af
  Resource Version:        19014474
  UID:                     b914ad18-2f5c-45cd-aa34-4ad7a2786536
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/1547...9301
  Dns Name:           mydomain.something
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt
  Key:      VqlmMCsb019CCFDggs03RyBLZ...nc767h_g.YfUcSfIKv...GxwfJMLWUGsB5kFKRc
  Solver:
    http01:
      Ingress:
        Class:  nginx
        Ingress Template:
          Metadata:
            Annotations:
              nginx.org/mergeable-ingress-type:  minion
        Service Type:                            ClusterIP
  Token:                                         VqlmMCsb019CC...03RyBLZJ0jo53LOiqnc767h_g
  Type:                                          HTTP-01
  URL:                                           https://acme-v02.api.letsencrypt.org/acme/chall-v3/15...49301/X--4pw
  Wildcard:                                      false
Events:                                          <none>

Any idea how I can solve this issue?

Update 1:

When I run curl http://some-domain.tld/.well-known/acme-challenge/VqlmMCsb019CC...gs03RyBLZJ0jo53LOiqnc767h_g in another pod, it returns:

curl: (52) Empty reply from server

When I do it locally (on my PC), it returns me the expected challenge-response.

-- Nrgyzer
cert-manager
kubernetes
lets-encrypt
ssl

2 Answers

8/25/2021

Try using dns01 challenge instead of HTTP-01

-- shiva
Source: StackOverflow

8/25/2021

Make sure your POD is returning something on the home URL or on the Home page of the domain that you are configuring on ingress host

You can also use the DNS-01 method for verification if HTTP-01 is not working

Here example for the DNS-01 :

Wild card certificate with cert-manager example

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: test123@gmail.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - selector:
        dnsZones:
          - "devops.example.in"
      dns01:
        route53:
          region: us-east-1
          hostedZoneID: Z0152EXAMPLE
          accessKeyID: AKIA5EXAMPLE
          secretAccessKeySecretRef:
            name: route53-secret
            key: secret-access-key
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: le-crt
spec:
  secretName: tls-secret
  issuerRef: 
    kind: Issuer
    name: letsencrypt-prod
  commonName: "*.devops.example.in"
  dnsNames:
    - "*.devops.example.in"
-- Harsh Manvar
Source: StackOverflow