Setting up Kubernetes - API not reachable from Pods

12/6/2018

I'm trying to setup a basic Kubernetes cluster on a (Ubuntu 16) VM. I've just followed the getting started docs and would expect a working cluster, but unfortunately, no such luck - no pods can't seem to connect to the Kubenernetes API. Since I'm new to Kubernetes it is very tough for me to find where things are going wrong. Provision script:

apt-get update && apt-get upgrade -y
apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl docker.io
apt-mark hold kubelet kubeadm kubectl
swapoff -a
sysctl net.bridge.bridge-nf-call-iptables=1
kubeadm init
mkdir -p /home/ubuntu/.kube
cp -i /etc/kubernetes/admin.conf /home/ubuntu/.kube/config
chown -R ubuntu:ubuntu /home/ubuntu/.kube
runuser -l ubuntu -c "kubectl apply -f \"https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')\""
runuser -l ubuntu -c "kubectl taint nodes --all node-role.kubernetes.io/master-"

Installation seems fine.

ubuntu@packer-Ubuntu-16:~$ kubectl get pods -o wide --all-namespaces
NAMESPACE     NAME                                       READY   STATUS             RESTARTS   AGE   IP               NODE               NOMINATED NODE   READINESS GATES
kube-system   coredns-86c58d9df4-lbp46                   0/1     CrashLoopBackOff   7          18m   10.32.0.2        packer-ubuntu-16   <none>           <none>
kube-system   coredns-86c58d9df4-t8nnn                   0/1     CrashLoopBackOff   7          18m   10.32.0.3        packer-ubuntu-16   <none>           <none>
kube-system   etcd-packer-ubuntu-16                      1/1     Running            0          17m   145.100.100.100  packer-ubuntu-16   <none>           <none>
kube-system   kube-apiserver-packer-ubuntu-16            1/1     Running            0          18m   145.100.100.100  packer-ubuntu-16   <none>           <none>
kube-system   kube-controller-manager-packer-ubuntu-16   1/1     Running            0          17m   145.100.100.100  packer-ubuntu-16   <none>           <none>
kube-system   kube-proxy-dwhhf                           1/1     Running            0          18m   145.100.100.100  packer-ubuntu-16   <none>           <none>
kube-system   kube-scheduler-packer-ubuntu-16            1/1     Running            0          17m   145.100.100.100  packer-ubuntu-16   <none>           <none>
kube-system   weave-net-sfvz5                            2/2     Running            0          18m   145.100.100.100  packer-ubuntu-16   <none>           <none>

Question: is it normal that the Kubernetes pods have as IP the ip of eth0 of the host (145.100.100.100)? Seems weird to me, I would expect them to have a virtual IP?

As you can see the coredns pod is crashing, because, well, it cannot reach the API.

This is as I understand it, the service:

ubuntu@packer-Ubuntu-16:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   22m

CoreDNS crashing, because API is unreachable:

ubuntu@packer-Ubuntu-16:~$ kubectl logs -n kube-system coredns-86c58d9df4-lbp46
.:53
2018-12-06T12:54:28.481Z [INFO] CoreDNS-1.2.6
2018-12-06T12:54:28.481Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
E1206 12:54:53.482269       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:318: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1206 12:54:53.482363       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:311: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
E1206 12:54:53.482540       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:313: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I tried launching a simple alpine pod/container. And indeed 10.96.0.1 doesn't responds to pings or anything else.

I'm stuck here. I've tried to google a lot but nothing comes up and my understanding is pretty basic. I guess something's up with the networking, but I don't know what (for me it seems suspicious that when doing get pods, the pods show up with the host IP, but perhaps this is normal also?)

-- Tommos
installation
kubernetes

1 Answer

12/6/2018

I found that the problem is caused by the host's iptables rules.

-- Tommos
Source: StackOverflow