My Kubernetes 3 VM cluster (1 controller, 2 workers on Ubuntu 20.04) is having problems with DNS host lookup. It cannot correctly resolve external host names. I found this trying to run Jenkins on my cluster, and Jenkins could not get its plugins on initial setup.

Can anyone shed light on the following?

When I am using this dnsutils pod, and shelling in to do an nslookup, it fails.

pod$ nslookup google.com
Server:		10.96.0.10
Address:	10.96.0.10#53

*** Can't find google.com.localdomain: No answer

However trying dig instead succeeds:

pod$ dig google.com
; <<>> DiG 9.11.6-P1 <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10886
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 272b157caff6a2b8 (echoed)
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		5	IN	A	142.250.191.174

;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Wed Apr 21 17:11:01 UTC 2021
;; MSG SIZE  rcvd: 77

If I use busybox, nslookup succeeds:

$ kubectl run curl-busybox --image=radial/busyboxplus:curl -i --tty --rm

pod$ nslookup google.com
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      google.com
Address 1: 2607:f8b0:4009:819::200e ord38s30-in-x0e.1e100.net
Address 2: 142.250.191.174 ord38s30-in-f14.1e100.net

The failing case appends cluster-only search suffixes as CoreDNS emits this in the log:

│ [INFO] 10.66.218.87:46775 - 8611 "A IN google.com.default.svc.cluster.local. udp 54 false 512" NXDOMAIN qr,aa,rd 147 0.00030851s                             │
│ [INFO] 10.66.218.87:55524 - 47795 "A IN google.com.cluster.local. udp 42 false 512" NXDOMAIN qr,aa,rd 135 0.000151907s                                       │
│ [INFO] 10.66.218.87:55367 - 64702 "AAAA IN google.com.localdomain. udp 40 false 512" NOERROR qr,aa 40 0.00094683s

The /etc/resolv.conf of dnsutils pod is:

search default.svc.cluster.local svc.cluster.local cluster.local localdomain
nameserver 10.96.0.10
options ndots:5

The succeeding dnsutils case with dig emits this:

[INFO] 10.66.218.87:59179 - 2071 "A IN google.com. udp 51 false 4096" NOERROR qr,rd,ra 54 0.000789572s   

My CoreDNS ConfigMap looks like this:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
           lameduck 5s
        }
        hosts /etc/coredns/customdomains.db cluster.dev {
          fallthrough
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
  customdomains.db: |
    192.168.149.130 kube-master.cluster.dev
kind: ConfigMap
metadata:
  creationTimestamp: "2021-04-16T17:22:52Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "396390"
  uid: 57853d1b-3675-4686-9abe-0185f20a5bc7