kube-dns CrashLoopBackOff

8/25/2018

I am following the https://github.com/kelseyhightower/kubernetes-the-hard-way

After completing the installation getting following issue:

kubectl get po —all-namespaces

kube-system kube-dns-598d7bf7d4-xfgfw 2/3 CrashLoopBackOff 10914 19d

kubectl logs -n kube-system kube-dns-598d7bf7d4-xfgfw -c kubedns

I0825 04:35:04.772663 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

I0825 04:35:05.272601 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

E0825 04:35:05.614968 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.32.0.1:443/api/v1/services?resourceVersion=0: x509: certificate is valid for 10.240.0.10, 10.240.0.11, 10.240.0.12, 35.240.168.187, 127.0.0.1, not 10.32.0.1

E0825 04:35:05.620146 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.32.0.1:443/api/v1/endpoints?resourceVersion=0: x509: certificate is valid for 10.240.0.10, 10.240.0.11, 10.240.0.12, 35.240.168.187, 127.0.0.1, not 10.32.0.1

I0825 04:35:05.772829 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

I0825 04:35:06.272745 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

E0825 04:35:06.620279 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.32.0.1:443/api/v1/services?resourceVersion=0: x509: certificate is valid for 10.240.0.10, 10.240.0.11, 10.240.0.12, 35.240.168.187, 127.0.0.1, not 10.32.0.1

E0825 04:35:06.626661 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.32.0.1:443/api/v1/endpoints?resourceVersion=0: x509: certificate is valid for 10.240.0.10, 10.240.0.11, 10.240.0.12, 35.240.168.187, 127.0.0.1, not 10.32.0.1

I0825 04:35:06.772930 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

I0825 04:35:07.272656 1 dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

E0825 04:35:07.628249 1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.32.0.1:443/api/v1/services?resourceVersion=0: x509: certificate is valid for 10.240.0.10, 10.240.0.11, 10.240.0.12, 35.240.168.187, 127.0.0.1, not 10.32.0.1

kubectl logs -n kube-system kube-dns-598d7bf7d4-xfgfw -c dnsmasq

I0825 04:34:59.855457 1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}

I0825 04:34:59.855908 1 nanny.go:94] Starting dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]

I0825 04:35:00.168181 1 nanny.go:119]

W0825 04:35:00.168488 1 nanny.go:120] Got EOF from stdout

I0825 04:35:00.168645 1 nanny.go:116] dnsmasq[8]: started, version 2.78 cachesize 1000

I0825 04:35:00.168792 1 nanny.go:116] dnsmasq[8]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify

I0825 04:35:00.168891 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain ip6.arpa

I0825 04:35:00.168986 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa

I0825 04:35:00.169077 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain cluster.local

I0825 04:35:00.169194 1 nanny.go:116] dnsmasq[8]: reading /etc/resolv.conf

I0825 04:35:00.169285 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain ip6.arpa

I0825 04:35:00.169395 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa

I0825 04:35:00.169484 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.1#10053 for domain cluster.local

I0825 04:35:00.169570 1 nanny.go:116] dnsmasq[8]: using nameserver 127.0.0.53#53

I0825 04:35:00.169818 1 nanny.go:116] dnsmasq[8]: read /etc/hosts - 7 addresses

kubectl logs -n kube-system kube-dns-598d7bf7d4-xfgfw -c sidecar

W0807 06:09:38.672889 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:47407->127.0.0.1:53: read: connection refused

W0807 06:09:43.673355 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:56792->127.0.0.1:53: read: connection refused

W0807 06:09:48.673754 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:39777->127.0.0.1:53: read: connection refused

W0807 06:09:53.674145 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:34342->127.0.0.1:53: read: connection refused

W0807 06:09:58.674551 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:52322->127.0.0.1:53: read: connection refused

W0807 06:10:03.674988 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:59016->127.0.0.1:53: read: connection refused

W0807 06:10:08.675349 1 server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:40075->127.0.0.1:53: read: connection refused

Further:

kubectl get svc kubernetes

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.32.0.1 443/TCP 19d

kubectl get endpoints kubernetes

NAME ENDPOINTS AGE
kubernetes 10.240.0.10:6443,10.240.0.11:6443,10.240.0.12:6443 19d

Please help on to solve the issue.

kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-13T22:29:03Z", GoVersion:"go1.9.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
-- Muthulingam
kube-dns
kubernetes

1 Answer

8/25/2018

You get the error:

x509: certificate is valid for 10.240.0.10, 10.240.0.11, 10.240.0.12, 35.240.168.187, 127.0.0.1, not 10.32.0.1

It sounds to me that you may have forgotten a certificate configuration for kube-apiserver.

Check all controller machines and if there's a missing parameter in services' scripts.

-- Nicola Ben
Source: StackOverflow