K
Q

Troubleshooting kube-dns pod : Timeout waiting for initialization

10/21/2018

My kube-dns pod is crashing:

kube-dns-6d4fc847dc-6bh59                               1/3     CrashLoopBackOff   5844       7d13h

These are the logs from the kubedns container

kubectl logs kube-dns-6d4fc847dc-6bh59 --namespace kube-system  -c kubedns

I1021 00:56:37.547936       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:38.047923       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
E1021 00:56:38.048390       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:192: Failed to list *v1.Service: Get https://100.64.0.1:443/api/v1/services?resourceVersion=0: dial tcp 100.64.0.1:443: i/o timeout
E1021 00:56:38.048405       1 reflector.go:201] k8s.io/dns/pkg/dns/dns.go:189: Failed to list *v1.Endpoints: Get https://100.64.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 100.64.0.1:443: i/o timeout
I1021 00:56:38.547922       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:39.047917       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:39.547923       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:40.047935       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:40.547928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:41.047931       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:41.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:42.047924       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:42.547931       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:43.047909       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:43.547921       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:44.047923       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:44.547932       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:45.047935       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:45.547927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:46.047928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:46.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:47.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:47.547933       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:48.047914       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:48.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:49.047927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:49.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:50.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:50.547928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:51.047933       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:51.547928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:52.047920       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:52.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:53.047913       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:53.547931       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:54.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:54.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:55.047928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:55.547938       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:56.047935       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:56.547937       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:57.047940       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:57.547941       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:58.047915       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:58.547934       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:59.047932       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:56:59.547937       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:00.047942       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:00.547934       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:01.047925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:01.547929       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:02.047915       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:02.547924       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:03.047906       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:03.547927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:04.047934       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:04.547927       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:05.047933       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:05.547925       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:06.047928       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:06.547926       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:07.047926       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
I1021 00:57:07.547930       1 dns.go:219] Waiting for [endpoints services] to be initialized from apiserver...
F1021 00:57:08.047911       1 dns.go:209] Timeout waiting for initialization

Kindly help.

-- smk
dns
kubernetes

Similar Questions

Kafka with Confluent Kubernetes Helm Charts = Schema Registry WakeupException
nginx ssl3_get_record wrong version number 502 Bad Gateway
Creating Persistent Volumes in Kubernetes on Google cloud platform
Encrypting secret data in kubernetes etcd store

1 Answer

10/21/2018

Looks like your kube-dns pods can't talk to your kube-apiserver. Port 443 on the kube-apiserver has been deprecated for a while. Your ConfigMap or config file is making it point to <kube-master:443>.

You can try overriding it on your kube-dns deployment with the --kube-master-url option. Something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7
        ...
        args:
        - --kube-master-url=https://<master-ip>:6443
        - --domain=cluster.local.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --v=2

You can get the from the output of:

$ kubectl cluster-info
-- Rico
Source: StackOverflow