K
Q

Can't see node after joining it to cluster - "getsockopt: connection refused"

5/14/2018

I am new in Kubernetes.

I have two nodes:

  • Master
  • Worker

I installed Kubernetes on both of them and on the muster I ran the kubeadm init... command in the Master node and received the command to join a new worker to the cluster:

sudo kubeadm join --token 61a503.3bdf2341a37a2732 192.168.190.159:6443 --discovery-token-ca-cert-hash sha256:ef66d8b7284af9e80f18767af39b5f164e00fd7fe714d3092e8ff682f07076da

I ran the above command inside the Worker node and it seems that it succeed:
enter image description here

This is the output:

[preflight] Running pre-flight checks.
    [WARNING FileExisting-crictl]: crictl not found in system path
[discovery] Trying to connect to API Server "192.168.190.159:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.190.159:6443"
[discovery] Requesting info from "https://192.168.190.159:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.190.159:6443"
[discovery] Successfully established connection with API Server "192.168.190.159:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

But when I go to the Master and run:

kubectl get nodes

I see only the master:

master@osboxes:~$ kubectl get nodes
NAME      STATUS    ROLES     AGE       VERSION
osboxes   Ready     master    4h        v1.9.1

Docker version on both nodes:

Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.6.2
 Git commit:   092cba3
 Built:        Thu Nov  2 20:40:23 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.6.2
 Git commit:   092cba3
 Built:        Thu Nov  2 20:40:23 2017
 OS/Arch:      linux/amd64
 Experimental: false

How can I find what is the problem?
Any idea what can it be ?
By the way, I tried the same thing on two nodes in AWS and it worked fine.

EDIT (15.5.2018): Logs
These are the logs from the kubelet daemon from the Worker node, I exported it with sudo journalctl -u kubelet > logs.txt

May 15 06:39:05 osboxes kubelet[12160]: E0515 06:39:05.113840   12160 kubelet_node_status.go:375] Unable to update node status: update node status exceeds retry count
May 15 06:39:06 osboxes kubelet[12160]: E0515 06:39:06.060871   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:06 osboxes kubelet[12160]: E0515 06:39:06.072458   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:06 osboxes kubelet[12160]: E0515 06:39:06.075082   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:07 osboxes kubelet[12160]: E0515 06:39:07.064412   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:07 osboxes kubelet[12160]: E0515 06:39:07.082627   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:07 osboxes kubelet[12160]: E0515 06:39:07.084203   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:08 osboxes kubelet[12160]: E0515 06:39:08.084848   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:08 osboxes kubelet[12160]: E0515 06:39:08.085296   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:08 osboxes kubelet[12160]: E0515 06:39:08.086186   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:09 osboxes kubelet[12160]: E0515 06:39:09.091850   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:09 osboxes kubelet[12160]: E0515 06:39:09.092907   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:09 osboxes kubelet[12160]: E0515 06:39:09.093494   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:10 osboxes kubelet[12160]: E0515 06:39:10.094472   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:10 osboxes kubelet[12160]: E0515 06:39:10.097289   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:10 osboxes kubelet[12160]: E0515 06:39:10.098355   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:11 osboxes kubelet[12160]: E0515 06:39:11.101260   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:11 osboxes kubelet[12160]: E0515 06:39:11.102788   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:11 osboxes kubelet[12160]: E0515 06:39:11.103772   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:12 osboxes kubelet[12160]: E0515 06:39:12.109494   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:12 osboxes kubelet[12160]: E0515 06:39:12.126419   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:12 osboxes kubelet[12160]: E0515 06:39:12.127858   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:13 osboxes kubelet[12160]: E0515 06:39:13.128797   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:13 osboxes kubelet[12160]: E0515 06:39:13.130811   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:13 osboxes kubelet[12160]: E0515 06:39:13.132159   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:14 osboxes kubelet[12160]: E0515 06:39:14.132703   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:14 osboxes kubelet[12160]: E0515 06:39:14.133885   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:14 osboxes kubelet[12160]: E0515 06:39:14.134534   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.123979   12160 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "osboxes": Get https://192.168.190.159:6443/api/v1/nodes/osboxes?resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.126886   12160 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "osboxes": Get https://192.168.190.159:6443/api/v1/nodes/osboxes: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.128832   12160 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "osboxes": Get https://192.168.190.159:6443/api/v1/nodes/osboxes: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.132161   12160 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "osboxes": Get https://192.168.190.159:6443/api/v1/nodes/osboxes: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.134043   12160 kubelet_node_status.go:383] Error updating node status, will retry: error getting node "osboxes": Get https://192.168.190.159:6443/api/v1/nodes/osboxes: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.134746   12160 kubelet_node_status.go:375] Unable to update node status: update node status exceeds retry count
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.142404   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.143773   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:15 osboxes kubelet[12160]: E0515 06:39:15.144730   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:16 osboxes kubelet[12160]: E0515 06:39:16.146062   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:16 osboxes kubelet[12160]: E0515 06:39:16.147948   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:16 osboxes kubelet[12160]: E0515 06:39:16.148963   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:17 osboxes kubelet[12160]: E0515 06:39:17.153690   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:17 osboxes kubelet[12160]: E0515 06:39:17.169648   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:17 osboxes kubelet[12160]: E0515 06:39:17.170775   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:18 osboxes kubelet[12160]: E0515 06:39:18.171909   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:18 osboxes kubelet[12160]: E0515 06:39:18.174020   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:18 osboxes kubelet[12160]: E0515 06:39:18.175013   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:19 osboxes kubelet[12160]: E0515 06:39:19.178296   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:19 osboxes kubelet[12160]: E0515 06:39:19.182903   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:19 osboxes kubelet[12160]: E0515 06:39:19.184147   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:20 osboxes kubelet[12160]: E0515 06:39:20.183063   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:20 osboxes kubelet[12160]: E0515 06:39:20.198007   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:20 osboxes kubelet[12160]: E0515 06:39:20.199996   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:21 osboxes kubelet[12160]: E0515 06:39:21.186122   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.190.159:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:21 osboxes kubelet[12160]: E0515 06:39:21.203974   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://192.168.190.159:6443/api/v1/nodes?fieldSelector=metadata.name%3Dosboxes&limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused
May 15 06:39:21 osboxes kubelet[12160]: E0515 06:39:21.207920   12160 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://192.168.190.159:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.190.159:6443: getsockopt: connection refused

Logs from Kubelet:

worker2@osboxes:~$ sudo kubelet
I0515 11:22:38.938557   33604 feature_gate.go:220] feature gates: &{{} map[]}
I0515 11:22:38.938712   33604 controller.go:114] kubelet config controller: starting controller
I0515 11:22:38.938757   33604 controller.go:118] kubelet config controller: validating combination of defaults and flags
W0515 11:22:38.949230   33604 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
I0515 11:22:38.987488   33604 server.go:182] Version: v1.9.1
I0515 11:22:38.987524   33604 feature_gate.go:220] feature gates: &{{} map[]}
I0515 11:22:38.987620   33604 plugins.go:101] No cloud provider specified.
W0515 11:22:38.987656   33604 server.go:328] standalone mode, no API client
W0515 11:22:39.028891   33604 server.go:236] No api server defined - no events will be sent to API server.
I0515 11:22:39.028997   33604 server.go:428] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
I0515 11:22:39.029367   33604 container_manager_linux.go:242] container manager verified user specified cgroup-root exists: /
I0515 11:22:39.029459   33604 container_manager_linux.go:247] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
I0515 11:22:39.029627   33604 container_manager_linux.go:266] Creating device plugin manager: false
W0515 11:22:39.032405   33604 kubelet_network.go:139] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0515 11:22:39.032456   33604 kubelet.go:571] Hairpin mode set to "hairpin-veth"
I0515 11:22:39.034512   33604 client.go:80] Connecting to docker on unix:///var/run/docker.sock
I0515 11:22:39.034568   33604 client.go:109] Start docker client with request timeout=2m0s
W0515 11:22:39.042224   33604 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
I0515 11:22:39.052530   33604 docker_service.go:232] Docker cri networking managed by kubernetes.io/no-op
I0515 11:22:39.075356   33604 docker_service.go:237] Docker Info: &{ID:N4M2:L4UZ:CZTV:LQHL:KAFZ:EB2Z:ZCF2:ED6G:KRR4:AI6X:KFQH:BTAH Containers:2 ContainersRunning:2 ContainersPaused:0 ContainersStopped:0 Images:7 Driver:aufs DriverStatus:[[Root Dir /var/lib/docker/aufs] [Backing Filesystem extfs] [Dirs 25] [Dirperm1 Supported true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host macvlan null overlay] Authorization:[] Log:[]} MemoryLimit:true SwapLimit:false KernelMemory:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:29 OomKillDisable:true NGoroutines:34 SystemTime:2018-05-15T11:22:39.055044415-04:00 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.13.0-36-generic OperatingSystem:Ubuntu 16.04.4 LTS OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc4206ae690 NCPU:1 MemTotal:2066481152 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:osboxes Labels:[] ExperimentalBuild:false ServerVersion:1.13.1 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:0xc420690500} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID: Expected:aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1} RuncCommit:{ID:N/A Expected:9df8b306d01f59d3a8029be411de015b7304dd8f} InitCommit:{ID:N/A Expected:949e6facb77383876aeff8a6944dde66b3089574} SecurityOptions:[name=apparmor name=seccomp,profile=default]}
I0515 11:22:39.075476   33604 docker_service.go:250] Setting cgroupDriver to cgroupfs
I0515 11:22:39.103735   33604 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
I0515 11:22:39.105235   33604 kuberuntime_manager.go:186] Container runtime docker initialized, version: 1.13.1, apiVersion: 1.26.0
I0515 11:22:39.114094   33604 server.go:755] Started kubelet
E0515 11:22:39.114169   33604 server.go:511] Starting health server failed: listen tcp 127.0.0.1:10248: bind: address already in use
E0515 11:22:39.114305   33604 kubelet.go:1275] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
W0515 11:22:39.114329   33604 kubelet.go:1359] No api server defined - no node status update will be sent.
I0515 11:22:39.114672   33604 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
I0515 11:22:39.115490   33604 server.go:129] Starting to listen on 0.0.0.0:10250
I0515 11:22:39.117147   33604 server.go:299] Adding debug handlers to kubelet server.
F0515 11:22:39.119054   33604 server.go:141] listen tcp 0.0.0.0:10250: bind: address already in use
-- E235
docker
kubernetes

Similar Questions

How to get browsable url from Docker-for-mac or Docker-for-Windows?
Cannot see volume content in a hostPath mount
How can I access the Kubernetes service from the Internet?
Kubernetes Django Disallowed Hosts
ingress-nginx controller install fail due to &quot;failed getting validating webhook&quot;

2 Answers

5/15/2018

OK, the problem was that both nodes (Master and Worker) had the same hostname :).
I noticed it after I ran kubectl describe node and in the Addresses field I saw the IP address of the worker with the same Hostname of the master:

Addresses:
  InternalIP:  192.168.190.162
  Hostname:    worker2node

I run sudo kubeadm reset on both Master and Worker.
On Master:

sudo swapoff -a
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.190.159 
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f kube-flannel.yml

On Worker:
Changed the hostname:

hostnamectl set-hostname worker2node
sudo vi /etc/hosts # (edit this file with the new name for 127.0.1.1)

Restart the worker and join it again.
I checked and now it was added.

-- E235
Source: StackOverflow
5/15/2018
 error reading /var/lib/kubelet/pki/kubelet.key, certificate and key must be supplied as a pair

Usually, this is a permission issue. Check permissions of the certificate file, it should be readable for Kubelet user.

If it does not help you - please share the Kubelet logs, namely the logs of the daemon, do not start it manually in the console.

Based on the update of the question:

192.168.190.159:6443: getsockopt: connection refused

It means that Kubelet on the node cannot connect to the master. Check the network connection between your node and the master. The node should be able to connect to https://192.168.190.159:6443, which is your API server endpoint.

-- Anton Kostenko
Source: StackOverflow