Kubernetes cert-manager cannot access letsencrypt API server

2/9/2020

I'm trying to setup cert-manager v0.13.0 on my minikube cluster. I've followed their tutorial, but it seems the cert-manager pod keeps timing out, trying to get to the LetsEncrypt API servers:

$ kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.13.0/deploy/manifests/00-crds.yaml
$ kubectl create namespace cert-manager
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm install cert-manager --namespace cert-manager --version v0.13.0 jetstack/cert-manager

Here's my acme yaml:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: xx@yyy.com
    privateKeySecretRef:
      name: my-issuer-account-key
    solvers:
      - dns01:
          cloudflare:
            email: xx@yyy.com
            apiKeySecretRef:
              name: cloudflare-api-token-secret
              key: api-token    

The cert-manager pod logs show the timeout:

I0209 20:43:34.382250       1 logger.go:90] Calling GetAccount
E0209 20:43:39.384093       1 setup.go:208] cert-manager/controller/clusterissuers "msg"="failed to verify ACME account" "error"="Get https://acme-staging-v02.api.letsencrypt.com/directory: dial tcp 192.64.119.254:443: i/o timeout" "related_resource_kind"="Secret" "related_resource_name"="my-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" 
E0209 20:43:39.385555       1 sync.go:81] cert-manager/controller/clusterissuers "msg"="error setting up issuer" "error"="Get https://acme-staging-v02.api.letsencrypt.com/directory: dial tcp 192.64.119.254:443: i/o timeout" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt" "resource_namespace"="" 
E0209 20:43:39.389659       1 controller.go:131] cert-manager/controller/clusterissuers "msg"="re-queuing item  due to error processing" "error"="Get https://acme-staging-v02.api.letsencrypt.com/directory: dial tcp 192.64.119.254:443: i/o timeout" "key"="letsencrypt" 

So I setup a bash pod to check the API's reachability, and there seems to be no problem:

$ kubectl run my-shell -n cert-manager --rm -i --tty --image ubuntu -- bash
$ apt-get update -y
$ apt-get install -y curl
$ https://acme-staging-v02.api.letsencrypt.org/directory

{
"xxxxxxxxx": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
"keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
"meta": {
    "caaIdentities": [
    "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
},
"newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}

Update: as requested, here's the /etc/resolve.conf file from the bash pod:

nameserver 10.96.0.10
search cert-manager.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

But I don't know how to get the same file from the cert-manager pod, as it doesn't let me open /bin/sh or /bin/bash.

I have no idea why the timeout occurs. Any thoughts?

-- Sagi Mann
cert-manager
kubernetes
lets-encrypt

1 Answer

2/18/2020

You mentioned acme server to acme-staging-v02.api.letsencrypt.org/directory, but it seems request is done to acme-staging-v02.api.letsencrypt.com/directory. There is a diff .com vs .org. Please check your clusterissuer yaml with the command:

kubectl get clusterissuer letsencrypt -o yaml

If you added wrong url on the yaml, you can always delete that clusterissuer, and create again.

-- Ahmad Faiyaz
Source: StackOverflow