nginx ingress with letsencrypt: Waiting for http-01 challenge propagation: wrong status code '404', expected '200'

3/20/2020

I'm trying to redeploy from GKE to Digital Ocean. I'm running into an issue with the challenge from letsencrypt. I believe K8s is telling me that the route cannot be found. The two hostnames/domains that letsencrypt is trying to do the challenge and failing for are CNAMES used by SendGrid. I'm not really sure where to start troubleshooting, my google-fu is failing me.

Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1alpha2
Kind:         Challenge
Metadata:
  Creation Timestamp:  2020-03-19T20:34:04Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  letsencrypt-certs-80407504-346698183
    UID:                   84ab9399-3a61-462e-a1c5-0831bd451a36
  Resource Version:        37060
  Self Link:               /apis/acme.cert-manager.io/v1alpha2/namespaces/default/challenges/letsencrypt-certs-80407504-346698183-813483524
  UID:                     ca14d996-3ecf-4dd8-8c53-057af7ae2b27
Spec:
  Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/3449670327
  Dns Name:   7502121.secodify.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-prod
  Key:      LiAawBR0bFRQfb2oXrvvNhph3ehQ-35lXJKkpjqgqb0.uWH4RnJfcABYba9T5b-QjoYnIw53rRtVhzsRIHIh39Y
  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        LiAawBR0bFRQfb2oXrvvNhph3ehQ-35lXJKkpjqgqb0
  Type:         http-01
  URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/3449670327/JGayWw
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:        <none>

my configmap looks like:

--- 
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {
      worker_connections  1024;
    }
    http {
      server {
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        listen 8080;
        server_name localhost;
        location ^~ /.well-known/acme-challenge/ {
          default_type "text/plain";
          rewrite /.well-known/acme-challenge/(.*) /$1 break;
        }
        location /static/ {
          autoindex on;    
          alias /code/core/static/; 
          include /etc/nginx/mime.types;
        }
        location = /favicon.ico {
        access_log off;
        log_not_found off;
        }
        location / {
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_pass http://127.0.0.1:8080/;
        }
      }
    }
---```
-- J. Brett Cunningham
kubernetes
kubernetes-ingress
lets-encrypt
nginx
nginx-ingress

2 Answers

3/20/2020

Add certmanager.k8s.io/cluster-issuer annotation to your Ingress configuration file. Similar problem: certmanager.

Moreover you have syntax mistake in server's location definition in ConfigMap:

 location ^~ /.well-known/acme-challenge/

Remember to add also rewrite and nginx class annotations to your Ingress configuration file.

-- MaggieO
Source: StackOverflow

3/20/2020

Turns out, the CNAMES were unused. After removing them from the config, I would do a kubectl get challenges again, but it was blank. This is because the challenges were already processed, I'm assuming, and therefore there were none pending.

I also had another issue where I forgot to make the relationship between the kubernetes service and the deployment pods.

Between the two issues, I got confused.

-- J. Brett Cunningham
Source: StackOverflow