Kubernetes DNS error listing services and endpoints

3/5/2016

I am getting an error with the DNS addon service on kubernetes.

If I run this command I see that the kube-dns service is being restarted:

kubectl get pods --namespace=kube-system -o wide

When I get the logs with:

kubectl logs kube-dns-v9-7mi17 -c kube2sky --namespace=kube-system

I get this being repeated many times:

E0305 04:39:30.837572       1 reflector.go:136] Failed to list *api.Endpoints: Get https://10.3.0.1:443/api/v1/endpoints: dial tcp 10.3.0.1:443: i/o timeout
E0305 04:39:30.948322       1 reflector.go:136] Failed to list *api.Service: Get https://10.3.0.1:443/api/v1/services: dial tcp 10.3.0.1:443: i/o timeout
E0305 04:40:01.838219       1 reflector.go:136] Failed to list *api.Endpoints: Get https://10.3.0.1:443/api/v1/endpoints: dial tcp 10.3.0.1:443: i/o timeout
E0305 04:40:01.948954       1 reflector.go:136] Failed to list *api.Service: Get https://10.3.0.1:443/api/v1/services: dial tcp 10.3.0.1:443: i/o timeout

The kubernetes service has that Virtual IP assigned, but the endpoint for kubernetes has the real IP of the service. Shouldn't the DNS service try to contact the API server using the endpoint IP instead of the virtual IP?

This is the definition I am using to create the DNS service:

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.3.0.10
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP

And this for the DNS replication controller:

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-dns-v9
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    version: v9
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-dns
    version: v9
  template:
    metadata:
      labels:
        k8s-app: kube-dns
        version: v9
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: etcd
        image: gcr.io/google_containers/etcd:2.0.9
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        command:
        - /usr/local/bin/etcd
        - -data-dir
        - /var/etcd/data
        - -listen-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -advertise-client-urls
        - http://127.0.0.1:2379,http://127.0.0.1:4001
        - -initial-cluster-token
        - skydns-etcd
        volumeMounts:
        - name: etcd-storage
          mountPath: /var/etcd/data
      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.11
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/kube2sky"
        - -domain=cluster.local
      - name: skydns
        image: gcr.io/google_containers/skydns:2015-03-11-001
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/skydns"
        - -machines=http://localhost:4001
        - -addr=0.0.0.0:53
        - -domain=cluster.local.
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 1
          timeoutSeconds: 5
      - name: healthz
        image: gcr.io/google_containers/exechealthz:1.0
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
        args:
        - -cmd=nslookup kubernetes.default.svc.cluster.local localhost >/dev/null
        - -port=8080
        ports:
        - containerPort: 8080
          protocol: TCP
      volumes:
      - name: etcd-storage
        emptyDir: {}
      dnsPolicy: Default
-- user1845791
kubernetes

1 Answer

6/1/2017

I get the same problem on DNS not being able to access the service list on Kubernetes v1.6.1. The problem was mention at github kubeadm issude #193.

I solved it with the latest comment in the previous mentioned issue:

2.Secondly, if your docker's version >=1.13, the default FORWARD chain policy was DROP, you should set default policy of the FORWARD chain to ACCEPT:

sudo iptables -P FORWARD ACCEPT
-- aitorhh
Source: StackOverflow