kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy

8/23/2017

Just started with kubernetes. I have three physical machines: 1 master, 2 nodes. I did basically getting started configuration. Everything seems up and running, nodes can communicate with master, but when I try to install a sample application (see in following kubernetes guide) I get this warning over and over again:

kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy.

And I can't access the app in: master_ip:30001

Any idea in what is going on and how to fix it?

Configuration

Here is the configuration:

$ kubectl get nodes

NAME                     STATUS    AGE       VERSION
master-precision-t1600   Ready     19h       v1.7.4
node2-precision-t1600    Ready     19h       v1.7.4

$ kubectl get pods --all-namespaces

NAMESPACE     NAME                                             READY     STATUS    RESTARTS   AGE
kube-system   etcd-master-precision-t1600                      1/1       Running   1          19h
kube-system   kube-apiserver-master-precision-t1600            1/1       Running   1          19h
kube-system   kube-controller-manager-master-precision-t1600   1/1       Running   1          19h
kube-system   kube-dns-2425271678-xrlp4                        3/3       Running   3          19h
kube-system   kube-proxy-58qm6                                 1/1       Running   1          19h
kube-system   kube-proxy-tjskl                                 1/1       Running   1          19h
kube-system   kube-scheduler-master-precision-t1600            1/1       Running   1          19h
kube-system   kubernetes-dashboard-3313488171-7n56j            1/1       Running   0          38m
kube-system   weave-net-1hjxl                                  2/2       Running   2          19h
kube-system   weave-net-lwk8r                                  2/2       Running   2          19h
sock-shop     carts-2469883122-h8f4n                           1/1       Running   0          1h
sock-shop     carts-db-1721187500-pkpk0                        1/1       Running   0          1h
sock-shop     catalogue-4293036822-hpkgp                       1/1       Running   0          1h
sock-shop     catalogue-db-1846494424-xlb8m                    1/1       Running   0          1h
sock-shop     front-end-2337481689-s8bkm                       1/1       Running   0          1h
sock-shop     orders-733484335-n7h4c                           1/1       Running   0          1h
sock-shop     orders-db-3728196820-12rt8                       1/1       Running   0          1h
sock-shop     payment-3050936124-kwqfs                         1/1       Running   0          1h
sock-shop     queue-master-2067646375-n8sgj                    1/1       Running   0          1h
sock-shop     rabbitmq-241640118-dqh6p                         1/1       Running   0          1h
sock-shop     shipping-2463450563-g01sw                        1/1       Running   0          1h
sock-shop     user-1574605338-kwqmp                            1/1       Running   0          1h
sock-shop     user-db-3152184577-w3f39                         1/1       Running   0          1h

$ kubectl describe nodes
Name:           master-precision-t1600
Role:           
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=master-precision-t1600
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:         node-role.kubernetes.io/master:NoSchedule
CreationTimestamp:  Tue, 22 Aug 2017 17:05:06 +0200
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  OutOfDisk         False   Wed, 23 Aug 2017 12:26:45 +0200     Tue, 22 Aug 2017 17:05:02 +0200     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Wed, 23 Aug 2017 12:26:45 +0200     Tue, 22 Aug 2017 17:05:02 +0200     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Wed, 23 Aug 2017 12:26:45 +0200     Tue, 22 Aug 2017 17:05:02 +0200     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Wed, 23 Aug 2017 12:26:45 +0200     Tue, 22 Aug 2017 17:05:02 +0200     KubeletReady            kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:   xxx.xxx.xxx.215
  Hostname: master-precision-t1600
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   8
 memory:                8127968Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    0
 cpu:                   8
 memory:                8025568Ki
 pods:                  110
System Info:
 Machine ID:            d718aa59fbe54581a9b058eb453ca453
 System UUID:           4C4C4544-005A-4410-805A-C4C04F32354A
 Boot ID:           687c603a-aad9-477a-a398-dfffeeaa4cd0
 Kernel Version:        4.10.0-32-generic
 OS Image:          Ubuntu 16.04.3 LTS
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.11.2
 Kubelet Version:       v1.7.4
 Kube-Proxy Version:        v1.7.4
ExternalID:         master-precision-t1600
Non-terminated Pods:        (8 in total)
  Namespace         Name                            CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                            ------------    ----------  --------------- -------------
  kube-system           etcd-master-precision-t1600             0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-apiserver-master-precision-t1600           250m (3%)   0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-controller-manager-master-precision-t1600      200m (2%)   0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-dns-2425271678-xrlp4               260m (3%)   0 (0%)      110Mi (1%)  170Mi (2%)
  kube-system           kube-proxy-58qm6                    0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           kube-scheduler-master-precision-t1600           100m (1%)   0 (0%)      0 (0%)      0 (0%)
  kube-system           kubernetes-dashboard-3313488171-7n56j           0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           weave-net-1hjxl                     20m (0%)    0 (0%)      0 (0%)      0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  830m (10%)    0 (0%)      110Mi (1%)  170Mi (2%)
Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  39m       30s     36  kubelet, master-precision-t1600         Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "kubernetes-dashboard-3313488171-7n56j_kube-system(1ed597d4-87e8-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.


Name:           node2-precision-t1600
Role:           
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=node2-precision-t1600
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:         <none>
CreationTimestamp:  Tue, 22 Aug 2017 17:10:43 +0200
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  OutOfDisk         False   Wed, 23 Aug 2017 12:26:49 +0200     Wed, 23 Aug 2017 11:42:43 +0200     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Wed, 23 Aug 2017 12:26:49 +0200     Wed, 23 Aug 2017 11:42:43 +0200     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      False   Wed, 23 Aug 2017 12:26:49 +0200     Wed, 23 Aug 2017 11:42:43 +0200     KubeletHasNoDiskPressure    kubelet has no disk pressure
  Ready         True    Wed, 23 Aug 2017 12:26:49 +0200     Wed, 23 Aug 2017 11:42:43 +0200     KubeletReady            kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:   129.241.110.167
  Hostname: node2-precision-t1600
Capacity:
 alpha.kubernetes.io/nvidia-gpu:    1
 cpu:                   8
 memory:                8127968Ki
 pods:                  110
Allocatable:
 alpha.kubernetes.io/nvidia-gpu:    1
 cpu:                   8
 memory:                8025568Ki
 pods:                  110
System Info:
 Machine ID:            d701c70173f547168978ca276552bb88
 System UUID:           4C4C4544-005A-4410-805A-B5C04F32354A
 Boot ID:           827de455-66cb-481d-a362-557a17db11f4
 Kernel Version:        4.10.0-32-generic
 OS Image:          Ubuntu 16.04.3 LTS
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.11.2
 Kubelet Version:       v1.7.4
 Kube-Proxy Version:        v1.7.4
ExternalID:         node2-precision-t1600
Non-terminated Pods:        (15 in total)
  Namespace         Name                    CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                    ------------    ----------  --------------- -------------
  kube-system           kube-proxy-tjskl            0 (0%)      0 (0%)      0 (0%)      0 (0%)
  kube-system           weave-net-lwk8r             20m (0%)    0 (0%)      0 (0%)      0 (0%)
  sock-shop         carts-2469883122-h8f4n          0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         carts-db-1721187500-pkpk0       0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         catalogue-4293036822-hpkgp      0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         catalogue-db-1846494424-xlb8m       0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         front-end-2337481689-s8bkm      100m (1%)   0 (0%)      100Mi (1%)  0 (0%)
  sock-shop         orders-733484335-n7h4c          0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         orders-db-3728196820-12rt8      0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         payment-3050936124-kwqfs        0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         queue-master-2067646375-n8sgj       0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         rabbitmq-241640118-dqh6p        0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         shipping-2463450563-g01sw       0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         user-1574605338-kwqmp           0 (0%)      0 (0%)      0 (0%)      0 (0%)
  sock-shop         user-db-3152184577-w3f39        0 (0%)      0 (0%)      0 (0%)      0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  120m (1%) 0 (0%)      100Mi (1%)  0 (0%)
Events:
  FirstSeen LastSeen    Count   From                    SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----                    -------------   --------    ------          -------
  44m       44m     1   kubelet, node2-precision-t1600              Normal      NodeReady       Node node2-precision-t1600 status is now: NodeReady
  44m       44m     3   kubelet, node2-precision-t1600              Normal      NodeHasSufficientDisk   Node node2-precision-t1600 status is now: NodeHasSufficientDisk
  44m       44m     3   kubelet, node2-precision-t1600              Normal      NodeHasSufficientMemory Node node2-precision-t1600 status is now: NodeHasSufficientMemory
  44m       44m     3   kubelet, node2-precision-t1600              Normal      NodeHasNoDiskPressure   Node node2-precision-t1600 status is now: NodeHasNoDiskPressure
  44m       44m     1   kubelet, node2-precision-t1600              Normal      NodeAllocatableEnforced Updated Node Allocatable limit across pods
  44m       44m     1   kubelet, node2-precision-t1600              Warning     Rebooted        Node node2-precision-t1600 has been rebooted, boot id: 827de455-66cb-481d-a362-557a17db11f4
  44m       44m     1   kubelet, node2-precision-t1600              Normal      Starting        Starting kubelet.
  44m       44m     1   kube-proxy, node2-precision-t1600           Normal      Starting        Starting kube-proxy.
  43m       11m     9   kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "rabbitmq-241640118-dqh6p_sock-shop(79e3bb08-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       5m      24  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "orders-db-3728196820-12rt8_sock-shop(79ca1e21-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       2m      22  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "payment-3050936124-kwqfs_sock-shop(79cb96f4-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       2m      28  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "shipping-2463450563-g01sw_sock-shop(79fa9dd4-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  43m       2m      22  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "carts-2469883122-h8f4n_sock-shop(79bbf964-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       1m      30  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "user-db-3152184577-w3f39_sock-shop(7a303582-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       1m      16  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "user-1574605338-kwqmp_sock-shop(7a11a937-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       1m      20  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "catalogue-db-1846494424-xlb8m_sock-shop(79c24789-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       54s     26  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "queue-master-2067646375-n8sgj_sock-shop(79d46bb2-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       45s     30  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "front-end-2337481689-s8bkm_sock-shop(79c49a6c-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  44m       24s     11  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "carts-db-1721187500-pkpk0_sock-shop(79bd1f99-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  43m       11s     25  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "catalogue-4293036822-hpkgp_sock-shop(79bf628c-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  43m       3s      25  kubelet, node2-precision-t1600              Warning     MissingClusterDNS   kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "orders-733484335-n7h4c_sock-shop(79c6f31c-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.
  43m       0s      228 kubelet, node2-precision-t1600              Warning     MissingClusterDNS   (combined from similar events): kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. pod: "payment-3050936124-kwqfs_sock-shop(79cb96f4-87e4-11e7-ab01-782bcba630bb)". Falling back to DNSDefault policy.

Edit 1:

As Mike told me in his answer I got the DNS IP with : kubectl get services --namespace=kube-system. Things I've tried so far to change the command-line flags:

    1. Add it in the ExecStart kubeadm.conf as --cluster-dns 10.96.0.10
    1. Execute kubelet --cluster-dns 10.96.0.10

Both without results

-- Jorge Cabrera
kubernetes

1 Answer

8/23/2017

The kubelet service needs a command-line flag to set the cluster DNS IP - it looks like you're running kube-dns, so you can get that IP by either running kubectl get services --namespace=kube-system or grabbing the IP from the "ClusterIP" field on the kube-dns service YAML or JSON config.

Once you have the IP, you'll have to set the --cluster-dns command-line flag for kubelet.

I haven't used kubeadm to setup a cluster, so I'm not sure how it runs the services and can't say how to change the command-line flags - hopefully somebody who knows can provide input for that piece.

-- Mike Cronce
Source: StackOverflow