I am getting an error with the DNS addon service on kubernetes.
If I run this command I see that the kube-dns service is being restarted:
kubectl get pods --namespace=kube-system -o wide
When I get the logs with:
kubectl logs kube-dns-v9-7mi17 -c kube2sky --namespace=kube-system
I get this being repeated many times:
E0305 04:39:30.837572 1 reflector.go:136] Failed to list *api.Endpoints: Get https://10.3.0.1:443/api/v1/endpoints: dial tcp 10.3.0.1:443: i/o timeout
E0305 04:39:30.948322 1 reflector.go:136] Failed to list *api.Service: Get https://10.3.0.1:443/api/v1/services: dial tcp 10.3.0.1:443: i/o timeout
E0305 04:40:01.838219 1 reflector.go:136] Failed to list *api.Endpoints: Get https://10.3.0.1:443/api/v1/endpoints: dial tcp 10.3.0.1:443: i/o timeout
E0305 04:40:01.948954 1 reflector.go:136] Failed to list *api.Service: Get https://10.3.0.1:443/api/v1/services: dial tcp 10.3.0.1:443: i/o timeout
The kubernetes service has that Virtual IP assigned, but the endpoint for kubernetes has the real IP of the service. Shouldn't the DNS service try to contact the API server using the endpoint IP instead of the virtual IP?
This is the definition I am using to create the DNS service:
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.3.0.10
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
And this for the DNS replication controller:
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-dns-v9
namespace: kube-system
labels:
k8s-app: kube-dns
version: v9
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-dns
version: v9
template:
metadata:
labels:
k8s-app: kube-dns
version: v9
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: etcd
image: gcr.io/google_containers/etcd:2.0.9
resources:
limits:
cpu: 100m
memory: 50Mi
command:
- /usr/local/bin/etcd
- -data-dir
- /var/etcd/data
- -listen-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -advertise-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -initial-cluster-token
- skydns-etcd
volumeMounts:
- name: etcd-storage
mountPath: /var/etcd/data
- name: kube2sky
image: gcr.io/google_containers/kube2sky:1.11
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/kube2sky"
- -domain=cluster.local
- name: skydns
image: gcr.io/google_containers/skydns:2015-03-11-001
resources:
limits:
cpu: 100m
memory: 50Mi
args:
# command = "/skydns"
- -machines=http://localhost:4001
- -addr=0.0.0.0:53
- -domain=cluster.local.
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 5
- name: healthz
image: gcr.io/google_containers/exechealthz:1.0
resources:
limits:
cpu: 10m
memory: 20Mi
args:
- -cmd=nslookup kubernetes.default.svc.cluster.local localhost >/dev/null
- -port=8080
ports:
- containerPort: 8080
protocol: TCP
volumes:
- name: etcd-storage
emptyDir: {}
dnsPolicy: Default
I get the same problem on DNS not being able to access the service list on Kubernetes v1.6.1. The problem was mention at github kubeadm issude #193.
I solved it with the latest comment in the previous mentioned issue:
2.Secondly, if your docker's version >=1.13, the default FORWARD chain policy was DROP, you should set default policy of the FORWARD chain to ACCEPT:
sudo iptables -P FORWARD ACCEPT