K
Q

Kubernetes: unreachable backend: read udp 10.200.0.9:46159->183.60.83.19:53: i/o timeout

3/16/2019

I use kubernetes v12, my system is ubuntu 16.

I use the followed command to create DNS resource.

wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed

wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh
bash deploy.sh -i 10.32.0.10 -r "10.32.0.0/24" -s -t coredns.yaml.sed | kubectl apply -f -

After created coredns resource: I check the resources status.

  1. check coredns service
root@master:~# kubectl get svc -n kube-system
NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
calico-typha   ClusterIP   10.32.0.10   <none>        5473/TCP   13h
  1. check code DNS pod endpoints
root@master:~# kubectl get ep -n kube-system
NAME                      ENDPOINTS   AGE
calico-typha              <none>      13h
kube-controller-manager   <none>      18d
kube-scheduler            <none>      18d
  1. My DNS config:
root@master:~# cat /etc/resolv.conf
nameserver 183.60.83.19
nameserver 183.60.82.98
  1. Check CoreDNS pod logs
root@master:~# kubectl get po -n kube-system | grep coredns-7bbd44c489-5thlj
coredns-7bbd44c489-5thlj   1/1     Running   0          13h
root@master:~#
root@master:~# kubectl logs -n kube-system pod/coredns-7bbd44c489-5thlj
.:53
2019-03-16T01:37:14.661Z [INFO] CoreDNS-1.2.6
2019-03-16T01:37:14.661Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = 2e2180a5eeb3ebf92a5100ab081a6381
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:45913->183.60.83.19:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:42500->183.60.82.98:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:48341->183.60.82.98:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:33007->183.60.83.19:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:52968->183.60.82.98:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:48992->183.60.82.98:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:35016->183.60.83.19:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:58058->183.60.82.98:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:51709->183.60.83.19:53: i/o timeout
 [ERROR] plugin/errors: 2 526217177044940556.1623766979909084596. HINFO: unreachable backend: read udp 10.200.0.93:53889->183.60.82.98:53: i/o timeout
root@master:~#

I found CoreDNS pod ip cannot connected to node DNS server ip address.

-- yang yang
calico
coredns
kubernetes
networking

Similar Questions

Kubernetes for securing service endpoints?
Several problems setting up Minikube on macOS
communication between GAE and GKE
Kubernetes Application Upgrade Strategy for stateful Apps
Kubernetes ingress controller not able to find the certificate secret
How to get CPU and memory usage of pod in percentage using promethus
Pod unable to curl localhost

2 Answers

3/19/2019

You should check calico firewall policy if it block internet access from pod. Another idea that you need to check which mode calico using: ipip / nat out going

-- nhancao
Source: StackOverflow
3/25/2019

You are lacking kube-dns service.

Input variable -i in deploy.sh sets IP for kube-dns service, and in your example, 10.32.0.10 is already assigned to calico-typha, you need to choose a different IP. Moreover, it should be in a valid range, but kubectl will prompt if it won't be.

You can always check it by running kubectl cluster-info dump | grep service-cluster-ip-range.

-- MWZ
Source: StackOverflow