Cannot get a kubernetes node to connect to master in Vagrant/Ansible

10/25/2016

I'm currently trying to learn Kubernetes by creating a Vagrant/Ansible project that allows me to bring up a master and node(s) and I've run into some problems. I'm currently use kubeadm to run the init and join function. For the init, I'm parsing the command line output to get the token (which I know is terrible; is there a more standard way to do this?):

---
  - command: kubeadm init
    args:
      creates: /etc/kubernetes/kubelet.conf
    register: kube_init

  - local_action: copy content={{ kube_init.stdout_lines[-1].split()[-2] }} dest=../kube.token
    when: kube_init is defined

  - local_action: copy content={{ kube_init.stdout_lines[-1].split()[-1] }} dest=../kube.master_ip
    when: kube_init is defined

Source: https://gitlab.com/djsumdog/skube/blob/0e25fca668e1067a8aa4af59b019411303838e86/ansible/roles/master/tasks/main.yml

But it works. The trouble is, I can't get the node to connect:

TASK: [node | command kubeadm join --token=1a3131.637de46bfdebadd4 10.0.2.15] *** 
failed: [node1] => {"changed": true, "cmd": ["kubeadm", "join", "--token=1a3131.637de46bfdebadd4", "10.0.2.15"], "delta": "0:00:00.079545", "end": "2016-10-25 01:54:51.858545", "rc": 1, "start": "2016-10-25 01:54:51.779000", "warnings": []}
stderr: error: <node/discovery> failed to request cluster info [Get http://10.0.2.15:9898/cluster-info/v1/?token-id=1a3131: dial tcp 10.0.2.15:9898: getsockopt: connection refused]
stdout: <util/tokens> validating provided token
<node/discovery> created cluster info discovery client, requesting info from "http://10.0.2.15:9898/cluster-info/v1/?token-id=1a3131"

FATAL: all hosts have already failed -- aborting

I can ping my master VM from the node VM and I can even ssh to it. It appears as if the kubernetes services are all running on the master too:

$ systemctl status kubelet.service 
kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Tue 2016-10-25 01:51:35 UTC; 13h ago
     Docs: http://kubernetes.io/docs/
 Main PID: 5726 (kubelet)
    Tasks: 15
   Memory: 45.5M
      CPU: 11min 37.179s
   CGroup: /system.slice/kubelet.service
           ├─5726 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-
           └─5740 journalctl -k -f

Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532171    5726 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-kubelet-pods-9d0e93cd\\x2d9a55\\x2d11e6\\x2db2dd\\x2d02558d159192-volumes-kubernetes.io\\x7esecret-default\\
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532175    5726 manager.go:843] ignoring container "/system.slice/var-lib-kubelet-pods-9d0e93cd\\x2d9a55\\x2d11e6\\x2db2dd\\x2d02558d159192-volumes-kubernetes.io\\x7esecret-default\\x2dtoken\\x2ddn3r9.m
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532184    5726 factory.go:104] Error trying to work out if we can handle /system.slice/run-docker-netns-default.mount: invalid container name
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532187    5726 factory.go:115] Factory "docker" was unable to handle container "/system.slice/run-docker-netns-default.mount"
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532190    5726 factory.go:108] Factory "systemd" can handle container "/system.slice/run-docker-netns-default.mount", but ignoring.
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532194    5726 manager.go:843] ignoring container "/system.slice/run-docker-netns-default.mount"
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532316    5726 factory.go:104] Error trying to work out if we can handle /system.slice/var-lib-docker-overlay-6d625f0d241bc6ae9eb243b430a71f078912a76147fc51e41e8e6e5f3226c6e9-merged.mount: error inspec
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532321    5726 factory.go:115] Factory "docker" was unable to handle container "/system.slice/var-lib-docker-overlay-6d625f0d241bc6ae9eb243b430a71f078912a76147fc51e41e8e6e5f3226c6e9-merged.mount"
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532325    5726 factory.go:108] Factory "systemd" can handle container "/system.slice/var-lib-docker-overlay-6d625f0d241bc6ae9eb243b430a71f078912a76147fc51e41e8e6e5f3226c6e9-merged.mount", but ignoring.
Oct 25 15:43:35 ubuntu-xenial kubelet[5726]: I1025 15:43:35.532330    5726 manager.go:843] ignoring container "/system.slice/var-lib-docker-overlay-6d625f0d241bc6ae9eb243b430a71f078912a76147fc51e41e8e6e5f3226c6e9-merged.mount"

Why is my master not listening for connections? Here is the exact commit I'm using:

https://gitlab.com/djsumdog/skube/tree/0e25fca668e1067a8aa4af59b019411303838e86

If you have ansible and vagrant installed, you should be able to run vagrant up and get the same result.

-- djsumdog
kubernetes

2 Answers

11/15/2016

It turns out this was due to my Vagrant setup. Both VMs were getting the same IP address. I needed to specify a :private_network and an ip: in my Vagrantfile:

...
config.vm.define "master" do |master|
  master.vm.network :private_network, ip: 10.10.30.10
...
...
config.vm.define "node1" do |node1|
  node1.vm.network :private_network, :ip: 10.10.30.50
...
-- djsumdog
Source: StackOverflow

11/10/2016

Try executing following commands after kubeadm init and before kubeadm join

kubectl taint nodes --all dedicated-
kubectl apply -f <add-on.yaml>

Refer this page http://kubernetes.io/docs/getting-started-guides/kubeadm/

-- Saloni Vithalani
Source: StackOverflow