Connection refused when using cert-manager to get a letsencrypt certificate for rancher

10/16/2020

I installed MicroK8s on my freshly installed Ubuntu 20.04 server using these instructions: https://microk8s.io/

Then I installed helm following these instructions: https://helm.sh/docs/intro/install/

And finally I installed rancher using these instructions: https://rancher.com/docs/rancher/v2.x/en/installation/install-rancher-on-k8s/

It appeared to work. The rancher pods are running, however rancher itself isn't reachable. There is no firewall active on the system. So I had a look at the logs:

kubectl -n cattle-system describe certificate

...
Spec:
  Dns Names:
    <domain>
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       rancher
  Secret Name:  tls-rancher-ingress
Status:
  Conditions:
    Last Transition Time:  2020-10-16T16:12:11Z
    Message:               Waiting for CertificateRequest "tls-rancher-ingress-3273743932" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  78m   cert-manager  Generated a new private key
  Normal  Requested     78m   cert-manager  Created new CertificateRequest resource "tls-rancher-ingress-3273743932"

kubectl -n cert-manager logs cert-manager-86b8b4f4b7-jvrg9

I1016 17:25:13.476038       1 sync.go:86] cert-manager/controller/issuers "msg"="Error initializing issuer: Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.53:53: read udp 127.0.0.1:34709-\u003e127.0.0.53:53: read: connection refused" "resource_kind"="Issuer" "resource_name"="rancher" "resource_namespace"="cattle-system"

I'm kind of new to kubernetes and this log isn't great. Where exactly is the connection getting refused? At the issuer (letsencrypt) itself? How do I fix it?

EDIT: Mainly I'm having trouble interpreting this last log message which is trying to tell me what's wrong. "lookup acme-v02.api.letsencrypt.org on 127.0.0.53:53" reads like it's incorrectly resolving that dns entry. If I ping it on the host, it correctly resolves it though. Is there some DNS config I need to do for kubernetes pods to be able to use the host's dns? I might also be entirely wrong here.

-- Andreas Hartmann
certificate
docker
kubernetes
kubernetes-helm
rancher

1 Answer

11/19/2020

Installing the microk8s dns and ingress addons solved the issue.

-- Andreas Hartmann
Source: StackOverflow