skydns is not able to resolve dns in kubernetes cluster

4/23/2016

I am setting up skydns for kubernetes following this template http://kubernetes.io/docs/getting-started-guides/docker-multinode/skydns.yaml.in. But it is not able to resolve dns. After looking for solutions also added -kube_master_url in kube2sky arguments but still the same issue. Here is the skydns logs:

2016/04/23 02:49:26 skydns: falling back to default configuration,  could not read from etcd: 501: All the given peers are not reachable (failed to propose on members [http://127.0.0.1:4001] twice [last error: Get http://127.0.0.1:4001/v2/keys/skydns/config?quorum=false&recursive=false&sorted=false: dial tcp 127.0.0.1:4001: connection refused]) [0]
2016/04/23 02:49:26 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
2016/04/23 02:49:26 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]
2016/04/23 02:49:33 skydns: error from backend: 501: All the given peers are not reachable (failed to propose on members [http://127.0.0.1:4001] twice [last error: Get http://127.0.0.1:4001/v2/keys/skydns/local/cluster/svc/default/kubernetes?quorum=false&recursive=true&sorted=false: dial tcp 127.0.0.1:4001: connection refused]) [0]

Any pointers? Kube2sky logs:

I0423 02:49:39.286489       1 kube2sky.go:436] Etcd server found:  http://127.0.0.1:4001
I0423 02:49:40.295909       1 kube2sky.go:503] Using http://172.17.0.1:8080 for kubernetes master
I0423 02:49:40.296183       1 kube2sky.go:504] Using kubernetes API v1
-- devd
kubernetes
skydns

1 Answer

5/2/2016

I had some iptables rules set which were blocking the connection on docker0 interface where kubernetes services including skydns were running. After flushing the rule it worked. Thus it appears that the first problem was with my local setup rather than skydns.

However the problem reappeared after installing local docker registry and I got this error:

I0427 20:30:45.183419 1 kube2sky.go:627] Ignoring error while waiting  for service default/kubernetes: Get https://10.0.0.1:443/api/v1/namespaces/default/services/kubernetes: x509: certificate signed by unknown authority. Sleeping 1s before retrying.

As a workaround I deleted the secrets using kubectl delete secrets/default-token-q4siz then restarted skydns and it started working again.

-- devd
Source: StackOverflow