K
Q

Kubernetes - unable to setup DNS

7/11/2017

After installing kubernetes according to: https://coreos.com/kubernetes/docs/latest/getting-started.html I am unable to setup dns addon.

Kubelet is running:

● kubelet.service
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-07-11 08:30:18 UTC; 35min ago
  Process: 5489 ExecStop=/usr/bin/rkt stop --uuid-file=/var/run/kubelet-pod.uuid (code=exited, status=0/SUCCESS)
  Process: 5507 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid (code=exited, status=0/SUCCESS)
  Process: 5503 ExecStartPre=/usr/bin/mkdir -p /var/log/containers (code=exited, status=0/SUCCESS)
  Process: 5501 ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 5519 (kubelet)
    Tasks: 21 (limit: 32768)
   Memory: 67.1M
      CPU: 40.541s
   CGroup: /system.slice/kubelet.service
           ├─5519 /kubelet --api-servers=http://127.0.0.1:8080 --register-schedulable=false --cni-conf-dir=/etc/kubernetes/cni/net.d --network-plugin= --container-runtime=docker --allow-privileged=true --pod-manifest-path=/etc/kubernetes/
           └─5574 journalctl -k -f

But kube-dns is unable to connect to api-server:

I0711 09:04:36.117052       1 dns.go:42] version: v1.6.0-alpha.0.680+3872cb93abf948-dirty
I0711 09:04:36.117449       1 server.go:107] Using https://10.3.0.1:443 for kubernetes master, kubernetes API: <nil>
I0711 09:04:36.118188       1 server.go:63] ConfigMap not configured, using values from command line flags
I0711 09:04:36.118223       1 server.go:113] FLAG: --alsologtostderr="false"
I0711 09:04:36.118234       1 server.go:113] FLAG: --config-map=""
I0711 09:04:36.118241       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0711 09:04:36.118246       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0711 09:04:36.118264       1 server.go:113] FLAG: --dns-port="10053"
I0711 09:04:36.118271       1 server.go:113] FLAG: --domain="cluster.local."
I0711 09:04:36.118277       1 server.go:113] FLAG: --federations=""
I0711 09:04:36.118289       1 server.go:113] FLAG: --healthz-port="8081"
I0711 09:04:36.118294       1 server.go:113] FLAG: --kube-master-url=""
I0711 09:04:36.118299       1 server.go:113] FLAG: --kubecfg-file=""
I0711 09:04:36.118303       1 server.go:113] FLAG: --log-backtrace-at=":0"
I0711 09:04:36.118310       1 server.go:113] FLAG: --log-dir=""
I0711 09:04:36.118315       1 server.go:113] FLAG: --log-flush-frequency="5s"
I0711 09:04:36.118322       1 server.go:113] FLAG: --logtostderr="true"
I0711 09:04:36.118327       1 server.go:113] FLAG: --stderrthreshold="2"
I0711 09:04:36.118331       1 server.go:113] FLAG: --v="0"
I0711 09:04:36.118335       1 server.go:113] FLAG: --version="false"
I0711 09:04:36.118342       1 server.go:113] FLAG: --vmodule=""
I0711 09:04:36.119467       1 server.go:155] Starting SkyDNS server (0.0.0.0:10053)
I0711 09:04:36.119543       1 server.go:167] Skydns metrics not enabled
I0711 09:04:36.119626       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0711 09:04:36.119644       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
E0711 09:05:06.120481       1 reflector.go:199] pkg/dns/dns.go:145: Failed to list *api.Endpoints: Get https://10.3.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.3.0.1:443: i/o timeout
E0711 09:05:06.120729       1 reflector.go:199] pkg/dns/dns.go:148: Failed to list *api.Service: Get https://10.3.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.3.0.1:443: i/o timeout

Logs from api server:

[restful] 2017/07/11 07:52:32 log.go:30: [restful/swagger] listing is available at https://10.10.0.201:443/swaggerapi/
[restful] 2017/07/11 07:52:32 log.go:30: [restful/swagger] https://10.10.0.201:443/swaggerui/ is mapped to folder /swagger-ui/
I0711 07:52:32.740429       1 serve.go:79] Serving securely on 0.0.0.0:443
I0711 07:52:32.740816       1 serve.go:94] Serving insecurely on 127.0.0.1:8080
I0711 07:52:33.679354       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication" (started 2017-07-11 07:52:32.965645672 +0000 UTC):
[28.85µs] [28.85µs] About to convert to expected version
[104.67µs] [75.82µs] Conversion done
[111.099µs] [6.429µs] About to store object in database
[713.576512ms] [713.465413ms] Object stored in database
[713.584455ms] [7.943µs] Self-link added
"Update /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication" [713.659549ms] [75.094µs] END
I0711 08:08:32.606430       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started 2017-07-11 08:08:31.933189525 +0000 UTC):
[8.907µs] [8.907µs] About to convert to expected version
[19.758µs] [10.851µs] Conversion done
[23.027µs] [3.269µs] About to store object in database
[673.145516ms] [673.122489ms] Object stored in database
[673.153091ms] [7.575µs] Self-link added
"Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" [673.202028ms] [48.937µs] END
I0711 08:20:10.566203       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started 2017-07-11 08:20:07.032156665 +0000 UTC):
[22.756µs] [22.756µs] About to convert to expected version
[40.481µs] [17.725µs] Conversion done
[44.63µs] [4.149µs] About to store object in database
[3.533855608s] [3.533810978s] Object stored in database
[3.53385947s] [3.862µs] Self-link added
"Update /api/v1/namespaces/kube-system/endpoints/kube-scheduler" [3.53395749s] [98.02µs] END
I0711 08:20:10.568558       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started 2017-07-11 08:20:08.786760574 +0000 UTC):
[12.63µs] [12.63µs] About to convert to expected version
[29.044µs] [16.414µs] Conversion done
[32.943µs] [3.899µs] About to store object in database
[1.781715989s] [1.781683046s] Object stored in database
[1.78171996s] [3.971µs] Self-link added
"Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" [1.781762523s] [42.563µs] END
I0711 08:20:48.669858       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started 2017-07-11 08:20:45.030964285 +0000 UTC):
[11.455µs] [11.455µs] About to convert to expected version
[25.762µs] [14.307µs] Conversion done
[29.427µs] [3.665µs] About to store object in database
[3.638826479s] [3.638797052s] Object stored in database
[3.638829618s] [3.139µs] Self-link added
"Update /api/v1/namespaces/kube-system/endpoints/kube-scheduler" [3.638855999s] [26.381µs] END
I0711 08:20:48.670137       1 trace.go:61] Trace "Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager" (started 2017-07-11 08:20:45.495019689 +0000 UTC):
[13.604µs] [13.604µs] About to convert to expected version
[33.752µs] [20.148µs] Conversion done
[37.844µs] [4.092µs] About to store object in database
[3.175049169s] [3.175011325s] Object stored in database

Api server ips:

# ip addr                                                                                                 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 5e:a7:f2:78:8d:04 brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.201/24 brd 10.10.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5ca7:f2ff:fe78:8d04/64 scope link 
       valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether ae:1e:17:85:1b:63 brd ff:ff:ff:ff:ff:ff
    inet 10.2.13.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::ac1e:17ff:fe85:1b63/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:06:e4:42:33 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

DNS Service:

kubectl --namespace=kube-system get svc -o wide
NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE       SELECTOR
kube-dns               10.3.0.10    <none>        53/UDP,53/TCP   1h        k8s-app=kube-dns
-- lisiecki.m
kube-dns
kubernetes

Similar Questions

GCLB : Frontend configuration : Error : Invalid value for field 'namedPorts[0].port
How to add websocket support to an ingress resource in Kubernetes on IBM Bluemix?
Issue while dynamically provisioning volumes on AWS, Kubernetes
Logging using Stackdriver API on Kubernetes / Google Container Engine (GKE)

1 Answer

7/11/2017

From what I see, the API server address (10.10.0.201) and the API server address the DNS service is trying to access (10.3.0.1) do not match. The DNS service must be configured to use 10.10.0.201 as the API address.

I think the --api-servers=http://127.0.0.1:8080 in Kubelet (I assume that the kube-dns service may be running in a different node from the api server) may be the cause of the issue.

-- Javier Salmeron
Source: StackOverflow