Certificate issued by cert manager reads as "issued by: cert-manager.local" instead of Let's Encrypt and does not work

10/2/2019

When I browse my website from Chrome, it says that the certificate is invalid, and if I check the details, this is what I see:

Issued to:
Common Name (CN)    test.x.example.com
Organization (O)    cert-manager
Organizational Unit (OU)    <Not Part Of Certificate>

Issued by:
Common Name (CN)    cert-manager.local
Organization (O)    cert-manager
Organizational Unit (OU)    <Not Part Of Certificate>

I don't understand what is going wrong. From cert-manager's output it would seem everything is going well:

I1002 15:56:52.761583       1 start.go:76] cert-manager "level"=0 "msg"="starting controller"  "git-commit"="95e8b7de" "version"="v0.9.1"
I1002 15:56:52.765337       1 controller.go:169] cert-manager/controller/build-context "level"=0 "msg"="configured acme dns01 nameservers" "nameservers"=["10.44.0.10:53"]
I1002 15:56:52.765777       1 controller.go:134] cert-manager/controller "level"=0 "msg"="starting leader election"
I1002 15:56:52.767133       1 leaderelection.go:235] attempting to acquire leader lease  cert-manager/cert-manager-controller...
I1002 15:56:52.767946       1 metrics.go:203] cert-manager/metrics "level"=0 "msg"="listening for connections on" "address"="0.0.0.0:9402"
I1002 15:58:18.940473       1 leaderelection.go:245] successfully acquired lease cert-manager/cert-manager-controller
I1002 15:58:19.043002       1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="challenges"
I1002 15:58:19.043050       1 base_controller.go:132] cert-manager/controller/challenges "level"=0 "msg"="starting control loop"
I1002 15:58:19.043104       1 controller.go:91] cert-manager/controller "level"=0 "msg"="not starting controller as it's disabled" "controller"="certificates-experimental"
I1002 15:58:19.043174       1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="orders"
I1002 15:58:19.043200       1 base_controller.go:132] cert-manager/controller/orders "level"=0 "msg"="starting control loop"
I1002 15:58:19.043376       1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="certificates"
I1002 15:58:19.043410       1 base_controller.go:132] cert-manager/controller/certificates "level"=0 "msg"="starting control loop"
I1002 15:58:19.043646       1 controller.go:91] cert-manager/controller "level"=0 "msg"="not starting controller as it's disabled" "controller"="certificaterequests-issuer-ca"
I1002 15:58:19.044292       1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="clusterissuers"
I1002 15:58:19.044459       1 base_controller.go:132] cert-manager/controller/clusterissuers "level"=0 "msg"="starting control loop"
I1002 15:58:19.044617       1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="ingress-shim"
I1002 15:58:19.044742       1 base_controller.go:132] cert-manager/controller/ingress-shim "level"=0 "msg"="starting control loop"
I1002 15:58:19.044959       1 controller.go:109] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="issuers"
I1002 15:58:19.045110       1 base_controller.go:132] cert-manager/controller/issuers "level"=0 "msg"="starting control loop"
E1002 15:58:19.082958       1 base_controller.go:91] cert-manager/controller/certificates/handleOwnedResource "msg"="error getting order referenced by resource" "error"="certificate.certmanager.k8s.io \"api-certificate\" not found" "related_resource_kind"="Certificate" "related_resource_name"="api-certificate" "related_resource_namespace"="staging" "resource_kind"="Order" "resource_name"="api-certificate-3031097725" "resource_namespace"="staging"
I1002 15:58:19.143501       1 base_controller.go:187] cert-manager/controller/orders "level"=0 "msg"="syncing item" "key"="staging/api-certificate-3031097725"
I1002 15:58:19.143602       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.143677       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.144011       1 sync.go:304] cert-manager/controller/orders "level"=0 "msg"="need to create challenges" "resource_kind"="Order" "resource_name"="api-certificate-3031097725" "resource_namespace"="staging" "number"=0
I1002 15:58:19.144043       1 logger.go:43] Calling GetOrder
I1002 15:58:19.144033       1 conditions.go:154] Setting lastTransitionTime for Certificate "cert-manager-webhook-webhook-tls" condition "Ready" to 2019-10-02 15:58:19.144027373 +0000 UTC m=+86.444394730
I1002 15:58:19.145112       1 conditions.go:154] Setting lastTransitionTime for Certificate "cert-manager-webhook-ca" condition "Ready" to 2019-10-02 15:58:19.145103359 +0000 UTC m=+86.445470721
I1002 15:58:19.145593       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="staging/api-certificate"
I1002 15:58:19.147411       1 issue.go:169] cert-manager/controller/certificates/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="api-certificate-3031097725" "related_resource_namespace"="staging"
I1002 15:58:19.148059       1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.148099       1 base_controller.go:187] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="staging/example-ingress"
I1002 15:58:19.148906       1 sync.go:71] cert-manager/controller/ingress-shim "level"=0 "msg"="not syncing ingress resource as it does not contain a \"certmanager.k8s.io/issuer\" or \"certmanager.k8s.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="example-ingress" "resource_namespace"="staging"
I1002 15:58:19.148925       1 base_controller.go:193] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="staging/example-ingress"
I1002 15:58:19.148133       1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.148963       1 conditions.go:91] Setting lastTransitionTime for Issuer "cert-manager-webhook-selfsign" condition "Ready" to 2019-10-02 15:58:19.148956891 +0000 UTC m=+86.449324275
I1002 15:58:19.149567       1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I1002 15:58:19.149759       1 conditions.go:91] Setting lastTransitionTime for Issuer "cert-manager-webhook-ca" condition "Ready" to 2019-10-02 15:58:19.149752693 +0000 UTC m=+86.450120071
I1002 15:58:19.148155       1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="default/letsencrypt-staging"
I1002 15:58:19.150457       1 setup.go:160] cert-manager/controller/issuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="default" "resource_kind"="Issuer" "resource_name"="letsencrypt-staging" "resource_namespace"="default"
I1002 15:58:19.148177       1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="staging/letsencrypt-staging-issuer"
I1002 15:58:19.148630       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="staging/api-certificate"
I1002 15:58:19.150669       1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="default/letsencrypt-staging"
I1002 15:58:19.151696       1 setup.go:160] cert-manager/controller/issuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging-secret-key" "related_resource_namespace"="staging" "resource_kind"="Issuer" "resource_name"="letsencrypt-staging-issuer" "resource_namespace"="staging"
I1002 15:58:19.151975       1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="staging/letsencrypt-staging-issuer"
I1002 15:58:19.153763       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.156512       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.157047       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.157659       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.158671       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.158827       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.171562       1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.172759       1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.173387       1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I1002 15:58:19.173465       1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.173562       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.174168       1 sync.go:329] cert-manager/controller/certificates/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="6905h41m20.825882558s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-webhook-tls" "related_resource_namespace"="cert-manager"
I1002 15:58:19.174487       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I1002 15:58:19.175092       1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.175489       1 base_controller.go:187] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.175743       1 base_controller.go:193] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-selfsign"
I1002 15:58:19.175978       1 base_controller.go:187] cert-manager/controller/certificates "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.176791       1 sync.go:329] cert-manager/controller/certificates/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="41945h41m15.823245228s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager"
I1002 15:58:19.177118       1 base_controller.go:193] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I1002 15:58:19.807942       1 base_controller.go:193] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="staging/api-certificate-3031097725"

Here is my configuration.

Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  tls:
  - hosts:
    - test.x.example.com
    secretName: letsencrypt-staging-certificate-secret
  rules:
  - host: test.x.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: example-frontend
          servicePort: 80

Issuer

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-staging-issuer
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: my-email@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging-secret-key
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01: {}

Certificate

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: test-x-example-com
spec:
  secretName: letsencrypt-staging-certificate-secret
  issuerRef:
    name: letsencrypt-staging-issuer
    kind: Issuer
  dnsNames:
    - test.x.example.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
        - test.x.example.com

Additional details: the secrets are in the staging namespace, like everything else except cert manager which is in the cert-manager namespace. The cluster is deployed on GKE.

EDIT: I'm wondering if it's possible that I hit the limits of the production environment in Let's Encrypt and got blocked. Is it possible to verify that somewhere?

-- rubik
cert-manager
google-cloud-platform
google-kubernetes-engine
kubernetes
lets-encrypt

1 Answer

10/3/2019

I finally solved the issue mostly by editing the Certificate configuration. I also switched from an Issuer to a ClusterIssuer but that should not have any impact on this issue. I think the problem was ACME verification.

Here is my new ClusterIssuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging-issuer
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: my-email@example.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging-secret-key
    # Enable the HTTP-01 challenge provider
    http01: {}

and, more importantly, the new Certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: test-x-example-com
spec:
  secretName: letsencrypt-staging-certificate-secret
  issuerRef:
    name: letsencrypt-staging-issuer
    kind: ClusterIssuer
  dnsNames:
    - test.x.example.com
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
        - test.x.example.com
-- rubik
Source: StackOverflow