Kubernetes Services reachable only on POD's host

9/11/2018

I have a 3-nodes bare metal cluster installed with KUBEADM (buildt with intel NUC running CentOS 7).

The master node is provided with 2 different network interfaces, one for external access and the second is configured to be the DHCP server of the cluster local network. IP forwarding and masquerading between the two netowrks is enabled through the following iptables rules

iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -i {{lan_interface_name}} -j ACCEPT iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -t nat -A POSTROUTING -o {{wan_interface_name}} -j MASQUERADE iptables -A FORWARD -i {{wan_interface_name}} -o {{lan_interface_name}} -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -I FORWARD -i {{lan_interface_name}} -o {{wan_interface_name}} -j ACCEPT

In addition I also enabled forwarding between docker interface and lan interface for every node in the cluster

iptables -I INPUT -i docker0 -j ACCEPT iptables -t nat -A POSTROUTING -o {{lan_interface_name}} -j MASQUERADE iptables -I FORWARD -i {{lan_interface_name}} -o docker0 -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -I FORWARD -i docker0 -o {{lan_interface_name}} -j ACCEPT

In order to be sure to have no iptables problems I enabled all the traffic by default

iptables -P OUTPUT ACCEPT iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT

While this is the /etc/sysctl.conf file

net.ipv4.ip_forward=1 net.bridge.bridge-nf-call-iptables=1

I followed the kubeadm instructions to launch a cluster with CALICO (https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) and everything seems to work properly except (even kube-dns is reachable from every POD in every node) for the exposed services.

Just for testing I created a nginx deployment and exposed it through nodeport: kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 30m nginx NodePort 10.96.227.49 <none> 80:31242/TCP 22m

If I try to curl the cluster IP within the same node of the POD everything is fine, while if I try to curl it in another node curl request timeout.

Any help would be really appreciated.

-- luke035
docker
kubeadm
kubernetes
networking
project-calico

1 Answer

9/11/2018

Kubernetes and Calico do convoluted things with iptables, so I believe you have a conflict or blocking rule somewhere. I recommend disabling all the rules to start then install docker and then your k8s with kubeadm.

-- Rico
Source: StackOverflow