I have an existing kubernetes 1.11.3 cluster and I'm unable to join new nodes to it.

I'm attempting to join by creating a new token with the --print-join-command flag through kubeadm on the master

Then I run that join command as sudo on the new worker node - but get the following output:

[discovery] Trying to connect to API Server "x.x.x.x:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://x.x.x.x:6443"
[discovery] Failed to request cluster info, will try again: [Unauthorized]
[discovery] Failed to request cluster info, will try again: [Unauthorized]

My understanding is that this output is caused by the --anonymous-auth flag on the api-server being set to false. But mine is explicitly set to true.

I'm unable to find any logs to give me more information about what api-server or controller manager are doing with the join request. Maybe I'm looking in the wrong place? Has anyone seen this before?

As requested - the output from kubeadm -v9 join (with the ip masked).

I0304 09:13:14.427364 3176882 join.go:226] [join] found NodeName empty
I0304 09:13:14.427443 3176882 join.go:227] [join] considered OS hostname as NodeName
[preflight] running pre-flight checks
I0304 09:13:14.427578 3176882 join.go:238] [preflight] running various checks on all nodes
I0304 09:13:14.427630 3176882 checks.go:253] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0304 09:13:14.427699 3176882 checks.go:291] validating the existence of file /etc/kubernetes/pki/ca.crt
I0304 09:13:14.427717 3176882 checks.go:291] validating the existence of file /etc/kubernetes/kubelet.conf
I0304 09:13:14.427732 3176882 checks.go:291] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0304 09:13:14.427753 3176882 kernelcheck_linux.go:45] validating the kernel module IPVS required exists in machine or not
    [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0304 09:13:14.431606 3176882 checks.go:138] validating if the service is enabled and active
I0304 09:13:14.448891 3176882 checks.go:340] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0304 09:13:14.448970 3176882 checks.go:340] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0304 09:13:14.449017 3176882 checks.go:653] validating whether swap is enabled or not
I0304 09:13:14.449083 3176882 checks.go:381] validating the presence of executable crictl
I0304 09:13:14.449131 3176882 checks.go:381] validating the presence of executable ip
I0304 09:13:14.449165 3176882 checks.go:381] validating the presence of executable iptables
I0304 09:13:14.449195 3176882 checks.go:381] validating the presence of executable mount
I0304 09:13:14.449224 3176882 checks.go:381] validating the presence of executable nsenter
I0304 09:13:14.449252 3176882 checks.go:381] validating the presence of executable ebtables
I0304 09:13:14.449281 3176882 checks.go:381] validating the presence of executable ethtool
I0304 09:13:14.449307 3176882 checks.go:381] validating the presence of executable socat
I0304 09:13:14.449346 3176882 checks.go:381] validating the presence of executable tc
I0304 09:13:14.449377 3176882 checks.go:381] validating the presence of executable touch
I0304 09:13:14.449402 3176882 checks.go:523] running all checks
I0304 09:13:14.451248 3176882 kernel_validator.go:81] Validating kernel version
I0304 09:13:14.451372 3176882 kernel_validator.go:96] Validating kernel config
I0304 09:13:14.780475 3176882 checks.go:411] checking whether the given node name is reachable using net.LookupHost
I0304 09:13:14.780711 3176882 checks.go:622] validating kubelet version
I0304 09:13:14.951219 3176882 checks.go:138] validating if the service is enabled and active
I0304 09:13:14.966615 3176882 checks.go:216] validating availability of port 10250
I0304 09:13:14.966764 3176882 checks.go:438] validating if the connectivity type is via proxy or direct
I0304 09:13:14.966809 3176882 join.go:251] [join] retrieving KubeConfig objects
[discovery] Trying to connect to API Server "*.*.*.*:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://*.*.*.*:6443"
I0304 09:13:14.967746 3176882 round_trippers.go:386] curl -k -v -XGET  -H "User-Agent: kubeadm/v1.11.3 (linux/amd64) kubernetes/a452946" -H "Accept: application/json, */*" 'https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0304 09:13:14.974195 3176882 round_trippers.go:405] GET https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 401 Unauthorized in 6 milliseconds
I0304 09:13:14.974229 3176882 round_trippers.go:411] Response Headers:
I0304 09:13:14.974238 3176882 round_trippers.go:414]     Content-Type: application/json
I0304 09:13:14.974250 3176882 round_trippers.go:414]     Content-Length: 129
I0304 09:13:14.974260 3176882 round_trippers.go:414]     Date: Mon, 04 Mar 2019 16:13:14 GMT
I0304 09:13:14.974302 3176882 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[discovery] Failed to request cluster info, will try again: [Unauthorized]
I0304 09:13:19.975419 3176882 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.3 (linux/amd64) kubernetes/a452946" 'https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0304 09:13:19.976262 3176882 round_trippers.go:405] GET https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 401 Unauthorized in 0 milliseconds
I0304 09:13:19.976288 3176882 round_trippers.go:411] Response Headers:
I0304 09:13:19.976300 3176882 round_trippers.go:414]     Content-Type: application/json
I0304 09:13:19.976313 3176882 round_trippers.go:414]     Content-Length: 129
I0304 09:13:19.976325 3176882 round_trippers.go:414]     Date: Mon, 04 Mar 2019 16:13:19 GMT
I0304 09:13:19.976366 3176882 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[discovery] Failed to request cluster info, will try again: [Unauthorized]