Why is kubelet using dns on IPv6 localhost [::1]:53?

1/27/2017

Kubelet isn't able to talk to the apiserver, because it can't resolve the apiserver's public DNS name. All it needs to do is use the host's resolv.conf file; but it's instead sending DNS requests to localhost. There's no DNS server (or cache) on localhost. From the kubelet logs:

Jan 27 22:10:42 kore4 kubelet-wrapper[1585]: E0127 22:10:42.583434    1585 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get https://ctrl1.example.com/api/v1/pods?fieldSelector=spec.nodeName%3Dkore4&resourceVersion=0: dial tcp: lookup ctrl1.example.com on [::1]:53: read udp [::1]:55253->[::1]:53: read: connection refused

Kubelet is being started like this (on CoreOS):

[Service]
Environment=KUBELET_VERSION=v1.5.2_coreos.0
Environment="RKT_OPTS=--uuid-file-save=/var/run/kubelet-pod.uuid \
  --volume dns,kind=host,source=/etc/resolv.conf \
  --mount volume=dns,target=/etc/resolv.conf \
  --volume var-log,kind=host,source=/var/log \
  --mount volume=var-log,target=/var/log \
  --volume cni-bin,kind=host,source=/opt/cni/bin \
  --mount volume=cni-bin,target=/opt/cni/bin"
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/usr/bin/mkdir -p /var/log/containers
ExecStartPre=/usr/bin/mkdir -p /opt/cni/bin
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid
ExecStart=/usr/lib/coreos/kubelet-wrapper \
  --api-servers=https://ctrl1.example.com,https://ctrl2.example.com,https://ctrl3.example.com \
  --cni-conf-dir=/etc/kubernetes/cni/net.d \
  --network-plugin=cni \
  --container-runtime=docker \
  --register-node=true \
  --allow-privileged=true \
  --pod-manifest-path=/etc/kubernetes/manifests \
  --cluster_dns=10.3.0.10 \
  --cluster_domain=cluster.local \
  --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
  --tls-cert-file=/etc/kubernetes/ssl/worker.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem
ExecStop=-/usr/bin/rkt stop --uuid-file=/var/run/kubelet-pod.uuid
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target

The host's /etc/resolv.conf is fine: It lists 2 local nameservers, plus 8.8.8.8 and 8.8.4.4, plus it has a search path.

-- Chris Jones
coreos
dns
kubernetes

0 Answers