Having some issue with my kubernetes cluster and DNS.
We recently updated to RHEL 7.5 and one of the machines was the Master. Once it back online most everything worked but I just noticed that external connections from the cluster do not resolve. Internal communication works great.
Here's the busybox nslookup results:
kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'kubernetes.default'
kubedns status
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations: <none>
Selector: k8s-app=kube-dns
Type: ClusterIP
IP: 10.96.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 10.32.0.18:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 10.32.0.18:53
Session Affinity: None
Events: <none>
All pods say they are up
NAME READY STATUS RESTARTS AGE
kube-dns-86f4d74b45-9m292 3/3 Running 26 44d
Lastest logs from the kubedns pods
I0523 15:59:14.291623 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0523 15:59:14.291638 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0523 15:59:14.791440 1 dns.go:170] Initialized services and endpoints from apiserver
I0523 15:59:14.791560 1 server.go:135] Setting up Healthz Handler (/readiness)
I0523 15:59:14.791579 1 server.go:140] Setting up cache handler (/cache)
I0523 15:59:14.791588 1 server.go:126] Status HTTP port 8081
I0523 16:48:05.175159 1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-controller-manager" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.236418 1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-dns" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.252139 1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-etcd" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.264292 1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-scheduler" in namespace "kube-system". DNS records will be created once endpoints show up.
I0524 13:04:38.008038 1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:48431->10.101.24.100:53: i/o timeout"
I0524 13:04:41.973900 1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:45738->10.101.24.100:53: i/o timeout"
I0524 17:26:54.056913 1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:52242->10.101.24.100:53: i/o timeout"
I0524 18:28:16.762002 1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:43051->10.101.24.100:53: i/o timeout"
I0524 18:28:20.762912 1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:57624->10.101.24.100:53: i/o timeout"
I am assuming this has something to do with the reboot.
Kubernetes: version 1.10.3
DNS: kube-dns
Network: Weavenet
Host OS: RHEL 7.5
Any help would be great.
Thanks
Not sure of the details of your upgrade, but check to see if your kube-dns version changed when you upgraded your OS. There’s an issue with kube-dns version 1.14.9 resolving external name services. You’ll need to roll back kube-dns to a previous build, or upgrade to 1.14.10.
See this issue: https://github.com/kubernetes/kops/issues/4986