Cert Manager Challenge Pending Kubernetes

9/26/2020

I have it working on one site application I already set up and now I am just trying to replicate the exact same thing for a different site/domain in another namespace.

So staging.correct.com is my working https domain

and staging.example.com is my not working https domain (http works - just not https)

When I do the following it shows 3 certs, the working one for correct and then 2 for the example.com when it should only have one for example:

kubectl get -A certificate

correct         staging-correct-com           True    staging-correct-com-tls   10d
example   staging-example-com           False   staging-example-com-tls   16h
example   staging-example-website-com   False   staging-example-com-tls   17h

When I do: kubectl get -A certificaterequests It shows 2 certificate requests for the example

example   staging-example-com-nl46v           False   15h
example   staging-example-website-com-plhqb   False   15h

When I do: kubectl get ingressroute -A

NAMESPACE       NAME                                         AGE
correct         correct-ingress-route                        10d
correct         correct-secure-ingress-route                 6d22h
kube-system     traefik-dashboard                            26d
example   example-website-ingress-route          15h
example   example-website-secure-ingress-route   15h
routing         dashboard                                    29d
routing         traefik-dashboard                            6d21h

When I do: kubectl get secrets -A (just showing the relevant ones)

correct           default-token-bphcm                       kubernetes.io/service-account-token 
correct           staging-correct-com-tls             kubernetes.io/tls 

example     default-token-wx9tx                       kubernetes.io/service-account-token   
example     staging-example-com-tls             Opaque                               
example     staging-example-com-wf224           Opaque                                
example     staging-example-website-com-rzrvw   Opaque   

Logs from cert manager pod:

1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="staging.example.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-bqjsj" "related_resource_namespace”=“example” "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="staging-example-com-ltjl6-1661100417-771202110" "resource_namespace”=“example” "resource_version"="v1" "type"="HTTP-01"

When I do: kubectl get challenge -A

example   staging-example-com-nl46v-1661100417-2848337980                   staging.example.com   15h
example   staging-example-website-com-plhqb-26564845-3987262508   pending   staging.example.com 

When I do: kubectl get order -A

NAMESPACE       NAME                                               STATE     AGE
example   staging-example-com-nl46v-1661100417         pending   17h
example   staging-example-website-com-plhqb-26564845   pending   17h

My yml files:

My ingress route:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  namespace: example
  name: example-website-ingress-route
annotations:
  kubernetes.io/ingress.class: "traefik"
  cert-manager.io/issuer: example-issuer-staging
  traefik.ingress.kubernetes.io/router.entrypoints: web
  traefik.frontend.redirect.entryPoint: https
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`staging.example.com`)
      middlewares:
        - name: https-only
      kind: Rule
      services:
        - name: example-website
          namespace: example
          port: 80

my issuer:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: example-issuer-staging
  namespace: example
spec:
  acme:
    # The ACME server  URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: example@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: staging-example-com-tls
    # Enable the HTTP-01 challenge  provider
    solvers:
    # An empty 'selector' means that this solver matches all domains
    - http01:
        ingress:
          class: traefik

my middleware:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-only
  namespace: example
spec:
  redirectScheme:
    scheme: https
    permanent: true

my secure ingress route:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  namespace: example
  name: example-website-secure-ingress-route
annotations:
  kubernetes.io/ingress.class: "traefik"
  cert-manager.io/issuer: example-issuer-staging
  traefik.ingress.kubernetes.io/router.entrypoints: websecure
  traefik.frontend.redirect.entryPoint: https
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`staging.example.com`)
      kind: Rule
      services:
        - name: example-website
          namespace: example
          port: 80
  tls:
    domains:                      
    - main: staging.example.com
    options:
      namespace: example
    secretName: staging-example-com-tls

my service:

apiVersion: v1
kind: Service
metadata:
  namespace: example
  name: 'example-website'
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 80
    - protocol: TCP
      name: https
      port: 443
      targetPort: 80
  selector:
    app: 'example-website'

my solver:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: staging-example-com
  namespace: example
spec:
  secretName: staging-example-com-tls
  issuerRef:
    name: example-issuer-staging
    kind: Issuer
  commonName: staging.example.com
  dnsNames:
  - staging.example.com

my app:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  namespace: example
  name: 'example-website'
  labels:
    app: 'example-website'
    tier: 'frontend'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: 'example-website'
  template:
    metadata:
      labels:
        app: 'example-website'
    spec:
      containers:
      - name: example-website-container
        image: richarvey/nginx-php-fpm:1.10.3 
        imagePullPolicy: Always
        env:
          - name: SSH_KEY
            value: 'secret'
          - name: GIT_REPO
            value: 'url of source code for site'
          - name: GIT_EMAIL
            value: 'example@example.com'
          - name: GIT_NAME
            value: 'example'
        ports:
        - containerPort: 80
  

How can I delete all these secrets, orders, certificates and stuff in the example namespace and try again? Does cert-manager let you do this without restarting them continuously?

EDIT:

I deleted the namespace and redeployed, then:

kubectl describe certificates staging-example-com -n example

Spec:
  Common Name:  staging.example.com
  Dns Names:
    staging.example.com
  Issuer Ref:
    Kind:       Issuer
    Name:       example-issuer-staging
  Secret Name:  staging-example-com-tls

Status:
  Conditions:
    Last Transition Time:        2020-09-26T21:25:06Z
    Message:                     Issuing certificate as Secret does not contain a certificate
    Reason:                      MissingData
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2020-09-26T21:25:07Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  staging-example-com-gnbl4

Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Issuing    3m10s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Reused     3m10s  cert-manager  Reusing private key stored in existing Secret resource "staging-example-com-tls"
  Normal  Requested  3m9s   cert-manager  Created new CertificateRequest resource "staging-example-com-qrtfx"

So then I did:

kubectl describe certificaterequest staging-example-com-qrtfx -n example

Status:
  Conditions:
    Last Transition Time:  2020-09-26T21:25:10Z
    Message:               Waiting on certificate issuance from order example/staging-example-com-qrtfx-1661100417: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready

Events:
  Type    Reason        Age    From          Message
  ----    ------        ----   ----          -------
  Normal  OrderCreated  8m17s  cert-manager  Created Order resource example/staging-example-com-qrtfx-1661100417
  Normal  OrderPending  8m17s  cert-manager  Waiting on certificate issuance from order example/staging-example-com-qrtfx-1661100417: ""

So I did:

kubectl describe challenges staging-example-com-qrtfx-1661100417 -n example

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending

Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    11m   cert-manager  Challenge scheduled for processing
  Normal  Presented  11m   cert-manager  Presented challenge using HTTP-01 challenge mechanism
-- Jacob
cert-manager
kubernetes
ssl-certificate
traefik

1 Answer

9/28/2020

I figured it out. The issue seems to be that IngressRoute (which is used in traefik) does not work with cert mananger. I just deployed this file, then the http check was confirmed, then I could delete it again. Hope this helps others with same issue.

Seems cert manager does support IngressRoute which is in Traefik? I opened the issue here so let's see what they say: https://github.com/jetstack/cert-manager/issues/3325

kubectl apply -f example-ingress.yml

File:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: example
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: "traefik"
    cert-manager.io/issuer: example-issuer-staging
spec:
  rules:
  - host: staging.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: example-website
          servicePort: 80
  tls:
  - hosts:
    - staging.example.com
    secretName: staging-example-com-tls
-- Jacob
Source: StackOverflow