K
Q

Coredns doesn't resolve time to time

6/18/2019

k8s: 1.13.1 corends: 1.2.6

I get "can't resolve" time to time trying to resolve service

[root@k8s-kube-master-vm01 ~]# kubectl exec -it  tiller-deploy-dc85f7fbd-vkqjs -n kube-system nslookup kubernetes-dashboard 
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes-dashboard
Address 1: 10.233.43.194
[root@k8s-kube-master-vm01 ~]# kubectl exec -it  tiller-deploy-dc85f7fbd-vkqjs -n kube-system nslookup kubernetes-dashboard 
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes-dashboard
Address 1: 10.233.43.194
[root@k8s-kube-master-vm01 ~]# kubectl exec -it  tiller-deploy-dc85f7fbd-vkqjs -n kube-system nslookup kubernetes-dashboard 
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'kubernetes-dashboard': Try again
command terminated with exit code 1

Coredns logs:

[root@k8s-kube-master-vm01 ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=coredns -o name); do kubectl logs --namespace=kube-system $p; done  | grep kubernetes-dashboard
2019-06-18T12:05:22.549Z [INFO] 10.233.124.35:41451 - 32323 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 161 0.000192773s
2019-06-18T12:05:22.55Z [INFO] 10.233.124.35:41451 - 32054 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 134 0.000112697s
2019-06-18T12:05:25.053Z [INFO] 10.233.124.35:41451 - 32323 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,rd,ra 161 0.000115262s
2019-06-18T12:05:25.053Z [INFO] 10.233.124.35:41451 - 32054 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,rd,ra 134 0.000037126s
2019-06-18T12:05:10.187Z [INFO] 10.233.124.35:50002 - 59429 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 161 0.000160883s
2019-06-18T12:05:10.187Z [INFO] 10.233.124.35:50002 - 59123 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 134 0.00006191s
2019-06-18T12:05:16.493Z [INFO] 10.233.124.35:40094 - 37873 "AAAA IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,rd,ra 161 0.000113602s
2019-06-18T12:05:16.493Z [INFO] 10.233.124.35:40094 - 37550 "A IN kubernetes-dashboard.kube-system.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd,ra 134 0.000154653s

External addresses are not resolved at all:

[root@k8s-kube-master-vm01 ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=coredns -o name); do kubectl logs --namespace=kube-system $p; done  | grep google.com
2019-06-18T10:18:31.888Z [INFO] 10.233.124.35:45081 - 28149 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000211934s
2019-06-18T10:18:31.888Z [INFO] 10.233.124.35:45081 - 27872 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000136701s
2019-06-18T10:18:34.391Z [INFO] 10.233.124.35:45081 - 28149 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000136248s
2019-06-18T10:18:34.391Z [INFO] 10.233.124.35:45081 - 27872 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000035216s
2019-06-18T10:20:23.899Z [INFO] 10.233.124.35:57585 - 35314 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000202737s
2019-06-18T10:20:23.899Z [INFO] 10.233.124.35:57585 - 35055 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,aa,rd,ra 139 0.000252218s
2019-06-18T10:20:26.4Z [INFO] 10.233.124.35:57585 - 35314 "AAAA IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000131456s
2019-06-18T10:20:26.4Z [INFO] 10.233.124.35:57585 - 35055 "A IN google.com.svc.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 0.000062335s
2019-06-18T12:11:18.188Z [INFO] 10.233.124.35:50407 - 11706 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000202781s
2019-06-18T12:11:18.188Z [INFO] 10.233.124.35:50407 - 11325 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.00015313s
2019-06-18T12:11:20.691Z [INFO] 10.233.124.35:50407 - 11706 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000148718s
2019-06-18T12:11:20.691Z [INFO] 10.233.124.35:50407 - 11325 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000124545s
2019-06-18T12:11:24.528Z [INFO] 10.233.124.35:54579 - 11413 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000074136s
2019-06-18T12:11:24.528Z [INFO] 10.233.124.35:54579 - 11083 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000033644s
2019-06-18T12:11:27.03Z [INFO] 10.233.124.35:54579 - 11413 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000136221s
2019-06-18T12:11:27.03Z [INFO] 10.233.124.35:54579 - 11083 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 0.000190085s
2019-06-18T10:18:31.894Z [INFO] 10.233.124.35:49521 - 63928 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000237032s
2019-06-18T10:18:31.895Z [INFO] 10.233.124.35:49521 - 63636 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000177646s
2019-06-18T10:20:23.905Z [INFO] 10.233.124.35:41078 - 52148 "AAAA IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000218841s
2019-06-18T10:20:23.906Z [INFO] 10.233.124.35:41078 - 51789 "A IN google.com.kube-system.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,aa,rd,ra 151 0.000090002s

I tried to view /etc/resolv.conf but there are no utilities in the coredns pod:

[root@k8s-kube-master-vm01 ~]# kubectl -n kube-system exec coredns-6fd7dbf94c-czjxj  -it sh
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
command terminated with exit code 126
[root@k8s-kube-master-vm01 ~]# kubectl -n kube-system exec coredns-6fd7dbf94c-czjxj  -it cat /etc/hosts
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"cat\": executable file not found in $PATH": unknown
command terminated with exit code 126
[root@k8s-kube-master-vm01 ~]# kubectl -n kube-system exec coredns-6fd7dbf94c-czjxj  -it /bin/cat /etc/hosts
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"/bin/cat\": stat /bin/cat: no such file or directory": unknown
command terminated with exit code 126
-- Andrey Belikov
coredns
kubernetes

Similar Questions

How to share configuration files between different clusters belonging to the same project in Google cloud Platform?
How to edit new project template to add quotas?
Kubernetes pod list somehow tied to computer?
NodePort service does not redirect request to another node
Updating containers of a Pod of a running system without messing up the application
Why is GKE HPA not scaling down?

2 Answers

6/18/2019

check that you enter the master IP and hostname in /etc/hosts

nano /etc/hosts
master YOUR_IP

and if does not help you try

# systemctl stop kubelet
# systemctl stop docker
# iptables --flush
# iptables -tnat --flush
# systemctl start kubelet
# systemctl start docker
-- yasin lachini
Source: StackOverflow
8/2/2019

Please start debugging your coredns pod and check if the DNS pod is running/restarting - following this examples:

for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done

Change coredns configmap and add log option:

kubectl -n kube-system edit configmap coredns

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
-- Hanx
Source: StackOverflow