K
Q

Question

cert-manager after major update stopped working

1/14/2020

The issue started after a major update of cert-manager from 0.6.0 to 0.11.0 version. The update has been processed via config backup, cert-manager remove, helm update, then cert-manager install and backup restore. No config changes during the update.

Pod and service are up, but no certs issued after update.

There are logs for cert-manager service:

 E0114 04:34:18.126497       1 sync.go:57] cert-manager/controller/ingress-shim "msg"="failed to determine issuer to be used for ingress resource" "error"="failed to determine issuer name to be used for ingress resource" "resource_kind"="Ingress" "resource_name"="ucb-sandbox-ingress" "resource_namespace"="cloud-engagement-sandbox" 
I0114 04:34:18.126791       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="cloud-engagement-sandbox/ucb-sandbox-ingress" 
I0114 04:34:18.127064       1 controller.go:129] cert-manager/controller/ingress-shim "level"=0 "msg"="syncing item" "key"="cloud-engagement-sandbox/ucf-sandbox-ingress" 
E0114 04:34:18.127294       1 sync.go:57] cert-manager/controller/ingress-shim "msg"="failed to determine issuer to be used for ingress resource" "error"="failed to determine issuer name to be used for ingress resource" "resource_kind"="Ingress" "resource_name"="ucf-sandbox-ingress" "resource_namespace"="cloud-engagement-sandbox" 
I0114 04:34:18.127534       1 controller.go:135] cert-manager/controller/ingress-shim "level"=0 "msg"="finished processing work item" "key"="cloud-engagement-sandbox/ucf-sandbox-ingress"

My ClusterIssuer yaml:

apiVersion: certmanager.k8s.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [removed]
    privateKeySecretRef:

      name: letsencrypt-prod
    http01: {}

And describe ClusterIssuer letsencrypt-prod

ClusterIssuer letsencrypt-prod
Name:         letsencrypt-prod
Namespace:
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"creationTimestamp":"2019-02-17T22:42:55Z"...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2019-02-17T22:42:55Z
  Generation:          1
  Resource Version:    53383155
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-prod
  UID:                 5e0c332f-3305-11e9-93cb-069443f5754c
Spec:
  Acme:
    Email:  [removed]
    Http 01:
    Private Key Secret Ref:
      Key:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:  https://acme-v02.api.letsencrypt.org/acme/acct/51694394
  Conditions:
    Last Transition Time:  2019-02-17T22:42:57Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

-- Nick Sinyakov

cert-manager

kubernetes

3 Answers

1/16/2020

It's sorted. The culprit was in 1) incomplete cert-manager install. 2) Also I've modified backup and replaced ALL certmanager.k8s.io with cert-manager.io and v1alpha1 with v1alpha2. 3) manually deleted other related to certmanager.k8s.io CRDs

-- Nick Sinyakov

Source: StackOverflow

1/16/2020

Thanks for reply. I removed old CRD after helm purge cert-manager and installed a fresh version 0.12 using manifests. My current CRD below:

kubectl get crd 
NAME                                    CREATED AT
certificaterequests.cert-manager.io     2019-11-01T01:37:03Z
certificates.cert-manager.io            2019-11-01T01:37:03Z
challenges.acme.cert-manager.io         2019-11-01T01:37:03Z
challenges.certmanager.k8s.io           2020-01-15T05:31:48Z
clusterissuers.cert-manager.io          2019-11-01T01:37:03Z
healthstates.azmon.container.insights   2019-08-29T10:13:59Z
issuers.cert-manager.io                 2019-11-01T01:37:03Z
orders.acme.cert-manager.io             2019-11-01T01:37:03Z
orders.certmanager.k8s.io               2020-01-15T05:31:49Z

And updated description of ClusterIssuer

kubectl describe ClusterIssuer letsencrypt-prod
Name:         letsencrypt-prod
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1alpha2
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2020-01-15T05:38:32Z
  Generation:          1
  Resource Version:    71299934
  Self Link:           /apis/cert-manager.io/v1alpha2/clusterissuers/letsencrypt-prod
  UID:                 4465c9ce-3759-11ea-be9c-0a7022c023e8
Spec:
  Acme:
    Email:  
    Private Key Secret Ref:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      Http 01:
        Ingress:
          Class:  nginx
      Selector:
Events:  <none>

I don't have ingress under cert-manager namespace. Also, my backup includes old certificates, CRD, Issuers, Certs and Certs requests etc but I don't know how to restore just what needed.

-- Nick Sinyakov

Source: StackOverflow

1/15/2020

The apiVersion has been changed from certmanager.k8s.io/v1alpha1 to cert-manager.io/v1alpha2. But You still have CRD with old apiVersion which you need to remove.

Follow below steps to upgrade cert manager paying attention to step 3 and 4.

1.Back up existing cert-manager resources, as per the backup and restore guide.

2.Uninstall cert-manager

3.Ensure the old cert-manager CRD resources have also been deleted: kubectl get crd | grep certmanager.k8s.io

4.Update the apiVersion on all your backed up resources from certmanager.k8s.io/v1alpha1 to cert-manager.io/v1alpha2.

5.Re-install cert-manager from scratch according to the installation guide

Here is the official upgrade guide

-- Arghya Sadhu

Source: StackOverflow

KQ

cert-manager after major update stopped working

Similar Questions

3 Answers

K
Q