Using jetstack/cert-manager-controller as cert-manager and google cloud dns for acme dns01 challenge, the version of cert-manager-controller is v1.6.0, the gcp sa store in secret google-clouddns-dns01-sa,

After create certificate, cert-manager pod show error "read udp 10.244.7.159:43347->192.168.20.31:53: read: connection refused"

10.244.7.159 => this is cert-manager pod ,

192.168.20.31:53 => I think this might be dns server for resolve domain, but after check , this is a node not related to dns or kubernetes cluster, I don't know why it appear here.

ClusterIssuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: google-clouddns-clusterissuer
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: google-clouddns-clusterissuer-key
    solvers:
    - selector: {}
      dns01:
        cloudDNS:
          project: "getbrightauto"
          serviceAccountSecretRef:
            name: google-clouddns-dns01-sa
            key: serviceAccountKey.json

Certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: getbrightauto-cert
  namespace: cert-manager
spec:
  dnsNames:
  - '*.getbrightauto.com'
  secretName: getbrightauto-com-tls
  issuerRef:
    kind: ClusterIssuer
    name: google-clouddns-clusterissuer

cert-manager pod error log

I1119 06:29:51.636448       1 conditions.go:201] Setting lastTransitionTime for Certificate "getbrightauto-cert" condition "Issuing" to 2021-11-19 06:29:51.636444946 +0000 UTC m=+9823.281270634
I1119 06:29:51.636440       1 conditions.go:201] Setting lastTransitionTime for Certificate "getbrightauto-cert" condition "Ready" to 2021-11-19 06:29:51.636401959 +0000 UTC m=+9823.281227652
I1119 06:29:51.710683       1 controller.go:161] cert-manager/controller/certificates-trigger "msg"="re-queuing item due to optimistic locking on resource" "key"="cert-manager/getbrightauto-cert" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"getbrightauto-cert\": the object has been modified; please apply your changes to the latest version and try again"
I1119 06:29:51.710896       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="cert-manager/getbrightauto-cert" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"
I1119 06:29:51.710938       1 conditions.go:201] Setting lastTransitionTime for Certificate "getbrightauto-cert" condition "Issuing" to 2021-11-19 06:29:51.710928378 +0000 UTC m=+9823.355754057
I1119 06:29:51.864458       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "getbrightauto-cert-9cq6z" condition "Approved" to 2021-11-19 06:29:51.86445152 +0000 UTC m=+9823.509277212
I1119 06:29:51.890542       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "getbrightauto-cert-9cq6z" condition "Ready" to 2021-11-19 06:29:51.890529646 +0000 UTC m=+9823.535355334
I1119 06:29:51.900851       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "getbrightauto-cert-9cq6z" condition "Ready" to 2021-11-19 06:29:51.900844807 +0000 UTC m=+9823.545670484
I1119 06:29:51.907289       1 controller.go:161] cert-manager/controller/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "key"="cert-manager/getbrightauto-cert-9cq6z" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"getbrightauto-cert-9cq6z\": the object has been modified; please apply your changes to the latest version and try again"
E1119 06:29:59.641932       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="read udp 10.244.7.159:43347->192.168.20.31:53: read: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"
E1119 06:29:59.660499       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="read udp 10.244.7.159:42189->192.168.20.31:53: read: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"
E1119 06:29:59.671445       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="read udp 10.244.7.159:57988->192.168.20.31:53: read: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"
E1119 06:30:19.881627       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="dial tcp 192.168.20.31:53: connect: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"
E1119 06:30:19.903955       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="read udp 10.244.7.159:56614->192.168.20.31:53: read: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"
I1119 06:30:19.913668       1 controller.go:161] cert-manager/controller/challenges "msg"="re-queuing item due to optimistic locking on resource" "key"="cert-manager/getbrightauto-cert-9cq6z-2473526240-1001066544" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"getbrightauto-cert-9cq6z-2473526240-1001066544\": the object has been modified; please apply your changes to the latest version and try again"
E1119 06:30:19.915387       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="read udp 10.244.7.159:57178->192.168.20.31:53: read: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"
E1119 06:30:19.926516       1 sync.go:186] cert-manager/controller/challenges "msg"="propagation check failed" "error"="read udp 10.244.7.159:46034->192.168.20.31:53: read: connection refused" "dnsName"="getbrightauto.com" "resource_kind"="Challenge" "resource_name"="getbrightauto-cert-9cq6z-2473526240-1001066544" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01"

kubectl  describe certificaterequests.cert-manager.io getbrightauto-cert-9cq6z -n cert-manager

OrderPending Waiting on certificate issuance from order cert-manager/getbrightauto-cert-9cq6z-24735
 
kubectl describe order getbrightauto-cert-9cq6z-2473526240 -n cert-manager
Status:
  Authorizations:
    Challenges:
      Token:        Un7TMPRu0UJDjPbqD-8Rd5pYDp1KUA02jmyBZG_fKl0
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/974003918/Y0LoTA
    Identifier:     getbrightauto.com
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/974003918
    Wildcard:       true
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/34217178/1056679278
  State:            pending
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/34217178/1056679278

Is this problem due to network setting? Where should I check? I had test a dnsutil pod to check in the same namespace which can resolve google.com to Ip.