I'm trying to get kubedns to work on my cluster but can't seem to get it to run. It keeps getting restarted because the healthz container keeps getting a failure on resolving kubernetes.default.svc.cluster.local.
Output of kubedns logs:
I0613 17:55:50.490883 1 dns.go:42] version: v1.6.0-alpha.0.680+3872cb93abf948-dirty
I0613 17:55:50.491791 1 server.go:107] Using https://k8s.internal for kubernetes master, kubernetes API: <nil>
I0613 17:55:50.691495 1 server.go:63] ConfigMap not configured, using values from command line flags
I0613 17:55:50.691539 1 server.go:113] FLAG: --alsologtostderr="false"
I0613 17:55:50.691564 1 server.go:113] FLAG: --config-map=""
I0613 17:55:50.691573 1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0613 17:55:50.691586 1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0613 17:55:50.691591 1 server.go:113] FLAG: --dns-port="10053"
I0613 17:55:50.691600 1 server.go:113] FLAG: --domain="cluster.local."
I0613 17:55:50.691612 1 server.go:113] FLAG: --federations=""
I0613 17:55:50.691618 1 server.go:113] FLAG: --healthz-port="8081"
I0613 17:55:50.691622 1 server.go:113] FLAG: --kube-master-url="https://k8s.internal"
I0613 17:55:50.691626 1 server.go:113] FLAG: --kubecfg-file="/etc/kubernetes/worker-kubeconfig.yaml"
I0613 17:55:50.691630 1 server.go:113] FLAG: --log-backtrace-at=":0"
I0613 17:55:50.691636 1 server.go:113] FLAG: --log-dir=""
I0613 17:55:50.691650 1 server.go:113] FLAG: --log-flush-frequency="5s"
I0613 17:55:50.691659 1 server.go:113] FLAG: --logtostderr="true"
I0613 17:55:50.691667 1 server.go:113] FLAG: --stderrthreshold="2"
I0613 17:55:50.691672 1 server.go:113] FLAG: --v="0"
I0613 17:55:50.691677 1 server.go:113] FLAG: --version="false"
I0613 17:55:50.691691 1 server.go:113] FLAG: --vmodule=""
I0613 17:55:50.691767 1 server.go:155] Starting SkyDNS server (0.0.0.0:10053)
I0613 17:55:50.691866 1 server.go:167] Skydns metrics not enabled
I0613 17:55:50.691967 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0613 17:55:50.691984 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0613 17:55:50.790263 1 server.go:126] Setting up Healthz Handler (/readiness)
I0613 17:55:50.791070 1 server.go:131] Setting up cache handler (/cache)
I0613 17:55:50.791096 1 server.go:120] Status HTTP port 8081
I0613 17:56:49.437868 1 server.go:150] Ignoring signal terminated (can only be terminated by SIGKILL)Output of healthz:
2017/06/13 17:53:09 Healthz probe on /healthz error: Result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local': Name does not resolve
, at 2017-06-13 17:53:08.890210169 +0000 UTC, error exit status 1The weird thing is if I exec that very same lookup against the kubedns container it succeeds:
$ kubectl --namespace kube-system exec kube-dns-v15-8nx4h -c kubedns -- nslookup kubernetes.default.svc.cluster.local localhost
Server: 127.0.0.1
Address 1: 127.0.0.1 localhost
Name: kubernetes.default.svc.cluster.local
Address 1: 172.17.0.1 kubernetes.default.svc.cluster.localBut on the dnsmasq container it doesn't:
$ kubectl --namespace kube-system exec kube-dns-v15-8nx4h -c dnsmasq -- nslookup kubernetes.default.svc.cluster.local localhost
nslookup: can't resolve 'kubernetes.default.svc.cluster.local': Name does not resolve
Server: 127.0.0.1
Address 1: 127.0.0.1 localhostI also don't see it do service discovery to add records, but I'm not sure if that's just because it doesn't have time to with the healthcheck killing it before it has the chance.
Here is the replication controller definition:
{
"apiVersion": "v1",
"kind": "ReplicationController",
"metadata": {
"labels": {
"k8s-app": "kube-dns",
"kubernetes.io/cluster-service": "true",
"version": "v15"
},
"name": "kube-dns-v15",
"namespace": "kube-system"
},
"spec": {
"replicas": 1,
"selector": {
"k8s-app": "kube-dns",
"version": "v15"
},
"template": {
"metadata": {
"labels": {
"k8s-app": "kube-dns",
"kubernetes.io/cluster-service": "true",
"version": "v15"
}
},
"spec": {
"volumes": [
{
"name": "ssl-certs",
"hostPath": {
"path": "/usr/share/ca-certificates"
}
},
{
"name": "kubeconfig",
"hostPath": {
"path": "/etc/kubernetes/worker-kubeconfig.yaml"
}
},
{
"name": "etc-kube-ssl",
"hostPath": {
"path": "/etc/kubernetes/ssl"
}
}
],
"containers": [
{
"volumeMounts": [
{
"mountPath": "/etc/ssl/certs",
"name": "ssl-certs",
"readOnly": true
},
{
"mountPath": "/etc/kubernetes/worker-kubeconfig.yaml",
"name": "kubeconfig",
"readOnly": true
},
{
"mountPath": "/etc/kubernetes/ssl",
"name": "etc-kube-ssl",
"readOnly": true
}
],
"args": [
"--domain=cluster.local.",
"--dns-port=10053",
"--kube-master-url=https://k8s.internal",
"--kubecfg-file=/etc/kubernetes/worker-kubeconfig.yaml"
],
"image": "gcr.io/google_containers/kubedns-amd64:1.9",
"livenessProbe": {
"failureThreshold": 5,
"httpGet": {
"path": "/healthz",
"port": 8080,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"successThreshold": 1,
"timeoutSeconds": 5
},
"name": "kubedns",
"ports": [
{
"containerPort": 10053,
"name": "dns-local",
"protocol": "UDP"
},
{
"containerPort": 10053,
"name": "dns-tcp-local",
"protocol": "TCP"
}
],
"readinessProbe": {
"httpGet": {
"path": "/readiness",
"port": 8081,
"scheme": "HTTP"
},
"initialDelaySeconds": 30,
"timeoutSeconds": 5
},
"resources": {
"limits": {
"cpu": "100m",
"memory": "200Mi"
},
"requests": {
"cpu": "100m",
"memory": "50Mi"
}
}
},
{
"args": [
"--cache-size=1000",
"--no-resolv",
"--server=127.0.0.1#10053"
],
"image": "gcr.io/google_containers/kube-dnsmasq-amd64:1.4.1",
"name": "dnsmasq",
"ports": [
{
"containerPort": 53,
"name": "dns",
"protocol": "UDP"
},
{
"containerPort": 53,
"name": "dns-tcp",
"protocol": "TCP"
}
]
},
{
"args": [
"-cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null",
"-port=8080",
"-quiet"
],
"image": "gcr.io/google_containers/exechealthz-amd64:v1.2.0",
"name": "healthz",
"ports": [
{
"containerPort": 8080,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "10m",
"memory": "20Mi"
},
"requests": {
"cpu": "10m",
"memory": "20Mi"
}
}
}
],
"dnsPolicy": "Default"
}
}
}
}