Certificate always in 'False' state using LetsEncrypt with cluster issuer in k8s

9/8/2021

I am unable to issue a working certificate for my ingress host in k8s. I use a ClusterIssuer to issue certificates and the same ClusterIssuer has issued certificates in the past for my ingress hosts under my domain name *xyz.com. But all of a sudden neither i can issue new Certificate with state 'True' for my host names nor a proper certificate secret (kubernetes.io/tls) gets created (but instead an Opaque secret gets created).

**strong text**

**kubectl describe certificate ingress-cert -n abc**

Name:         ingress-cert
Namespace:    abc
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1beta1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2021-09-08T07:48:32Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  test-ingress
    UID:                   c03ffec0-df4f-4dbb-8efe-4f3550b9dcc1
  Resource Version:        146643826
  Self Link:               /apis/cert-manager.io/v1beta1/namespaces/abc/certificates/ingress-cert
  UID:                     90905ab7-22d2-458c-b956-7100c4c77a8d
Spec:
  Dns Names:
    abc.xyz.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       letsencrypt
  Secret Name:  ingress-cert
Status:
  Conditions:
    Last Transition Time:        2021-09-08T07:48:33Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2021-09-08T07:48:33Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  ingress-cert-gdq7g
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    11m   cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  11m   cert-manager  Stored new private key in temporary Secret resource "ingress-cert-gdq7g"
  Normal  Requested  11m   cert-manager  Created new CertificateRequest resource "ingress-cert-dp6sp"

I checked the certificate request and it contains no events. Also i can see no challenges. I have added the logs below. Any help would be appreciated

kubectl describe certificaterequest ingress-cert-dp6sp -n abc

Namespace:    abc
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: ingress-cert
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: ingress-cert-gdq7g
API Version:  cert-manager.io/v1beta1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2021-09-08T07:48:33Z
  Generate Name:       ingress-cert-
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  ingress-cert
    UID:                   90905ab7-22d2-458c-b956-7100c4c77a8d
  Resource Version:        146643832
  Self Link:               /apis/cert-manager.io/v1beta1/namespaces/abc/certificaterequests/ingress-cert-dp6sp
  UID:                     fef72617-fc1d-4384-9f4b-a7e4502582d8
Spec:
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2Z6Q0NBV2NDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxMNgphTGhZNjhuNnhmMUprYlF5ek9OV1J4dGtLOXJrbjh5WUtMd2l4ZEFMVUl0TERra0t6Uksyb3loZzRMMThSQmQvCkNJaGJ5RXBYNnlRditKclRTOC84T1A0MWdwTUxBLzROdVhXWWtyeWhtZFdNaFlqa21OOFpiTUk1SlZZcVV2cVkKRWQ1b2cydmVmSjU1QlJPRExsd0o3YjBZa3hXckUwMGJxQ1ExWER6ZzFhM08yQ2JWd1NQT29WV2x6Uy9CdzRYVgpMeVdMS3E4QU52b2dZMUxXRU8xcG9YelRObm9LK2U2YVZueDJvQ1ZLdGxPaG1iYXRHYXNSaTJKL1FKK0dOWHovCnFzNXVBSlhzYVErUzlxOHIvbmVMOXNPYnN2OWd1QmxCK09yQVg2eHhkNHZUdUIwVENFU00zWis2c2MwMFNYRXAKNk01RlY3dkFFeDQyTWpuejVoa0NBd0VBQWFBNk1EZ0dDU3FHU0liM0RRRUpEakVyTUNrd0p3WURWUjBSQkNBdwpIb0ljY25kemMyZHdMbU5zYjNWa1oyRjBaUzV0YVdOeWIyWnBiaTVrWlRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUFTQ0cwTXVHMjZRbVFlTlBFdmphNHZqUUZOVFVINWVuMkxDcXloY2ZuWmxocWpMbnJqZURuL2JTV1hwdVIKTnhXTnkxS0EwSzhtMG0rekNPbWluZlJRS1k2eHkvZU1WYkw4dTgrTGxscDEvRHl3UGxvREE2TkpVOTFPaDM3TgpDQ0E4NWphLy9FYVVvK0p5aHBzaTZuS1d4UXRpYXdmYXhuNUN4SENPWGF5Qzg0Q0IzdGZ2WWp6YUF3Ykx4akxYCmxvd09LUHNxSE51ZktFM0NtcjZmWGgramd5VWhxamYwOUJHeGxCWEFsSVNBNkN5dzZ2UmpWamFBOW82TmhaTXUKbmdheWZON00zUzBBYnAzVFFCZW8xYzc3QlFGaGZlSUE5Sk51SWtFd3EvNXppYVY1RDErNUxSSnR5ZkVpdnJLTwpmVjQ5WkpCL1BGOTdiejhJNnYvVW9CSkc2Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
Status:
  Conditions:
    Last Transition Time:  2021-09-08T07:48:33Z
    Message:               Waiting on certificate issuance from order abc/ingress-cert-dp6sp-3843501305: ""
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:                    <none>

Here is the ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 20m
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  rules:
    - host: abc.xyz.com
      http:
        paths:
          - path: /static
            backend:
              serviceName: app-service
              servicePort: 80
          - path: /
            backend:
              serviceName: app-service
              servicePort: 8000
  tls:
  - hosts:
    - abc.xyz.com
    secretName: ingress-cert

Here is the clusterissuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: example@user.de
    privateKeySecretRef:
      name: letsencrypt-key
    solvers:
    - http01:
        ingress:
          class: nginx
-- devcloud
acme
kubernetes
kubernetes-ingress
lets-encrypt

1 Answer

9/8/2021

Ideally your ingress pointing to the secret which is storing the secret or SSL/TLS key cert.

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 20m
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  rules:
    - host: abc.xyz.com
      http:
        paths:
          - path: /static
            backend:
              serviceName: app-service
              servicePort: 80
          - path: /
            backend:
              serviceName: app-service
              servicePort: 8000
  tls:
  - hosts:
    - abc.xyz.com
    secretName: letsencrypt-key

Your cluster issue storing the key

privateKeySecretRef:
      name: letsencrypt-key

You have to use this secret and attach this to ingress.

If secret already storing cert with a domain

test.example.com and you are trying to get a new cert with hello.example.com

in this case using cluster issuer will overwrite the secret and might loss old cert stored inside secret.

You can create the multiple clusterissuer,

One storing and connect to single ingress, first.example.com

Second cluster issuer with different key name

privateKeySecretRef:
      name: letsencrypt-key

and new key or secret will get attached to the ingress.

-- Harsh Manvar
Source: StackOverflow