I am evaluating Kubernetes and installed it following this tutorial on CentOS 7.2 box and VMware VMs. My environment has one Docker Registry, one Etcd server, one Kube Master and two Kube Nodes. Every single piece of the communication works like a charm, thanks to Flannel and the excellent Kubernetes docs and papers out there. One exception: the communication between one Node to the service's clusterIP.
Some information regarding my environment:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE NODE
nginx-a2klb 1/1 Running 2 1d 10.200.81.54
$ kubectl get pods/nginx-a2klb -o yaml | grep podIP
podIP: 10.252.54.2
$ kubectl get svc/nginx -o yaml | grep clusterIP
clusterIP: 10.254.0.7Trying the communication from 10.200.81.54 (pod's Node):
$ curl http://10.254.0.7
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
From 10.200.81.53 (the other Node):
$ curl http://10.252.54.2 # podIP
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
$ curl http://10.254.0.7 # clusterIP
curl: (7) Failed connect to 10.254.0.7:80; Connection timed out
Long story short: inside containers the communication always works, from the same node or from other nodes. Using the podIP the communication always works as well. ExternalIP or NodePort always works, even outside the cluster. Using the clusterIP the communication only work from the same VM where the only pod is running, failing from other Nodes or from the Kube Master. Is this behavior expected? What can I do to analyze and fix this?
Update1
kubeletproxy, apiserver, scheduler, etc) run inside privileged containers via hyperkubeUpdate2
The relevant part of iptables (note that pod and service IPs changed since the first post):
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.252.36.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4d415351 -j MASQUERADE
-A KUBE-SEP-722QG7UQNTPWDFBY -s 10.200.81.52/32 -m comment --comment "default/kubernetes:" -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-722QG7UQNTPWDFBY -p tcp -m comment --comment "default/kubernetes:" -m tcp -j DNAT --to-destination 10.200.81.52:443
-A KUBE-SEP-7F6QLYX4EVXMHVGW -s 10.252.45.6/32 -m comment --comment "default/ng:" -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-7F6QLYX4EVXMHVGW -p tcp -m comment --comment "default/ng:" -m tcp -j DNAT --to-destination 10.252.45.6:80
-A KUBE-SEP-SFIQGU7OZTZRBGQ6 -s 10.252.45.2/32 -m comment --comment "kube-system/kube-dns:dns" -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-SFIQGU7OZTZRBGQ6 -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.252.45.2:53
-A KUBE-SEP-WT6RQUWXRXGAUOJF -s 10.252.45.2/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j MARK --set-xmark 0x4d415351/0xffffffff
-A KUBE-SEP-WT6RQUWXRXGAUOJF -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.252.45.2:53
-A KUBE-SERVICES -d 10.254.0.114/32 -p tcp -m comment --comment "default/ng: cluster IP" -m tcp --dport 80 -j KUBE-SVC-LYRG26ZZO4GOQOI3
-A KUBE-SERVICES -d 10.254.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.254.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes: cluster IP" -m tcp --dport 443 -j KUBE-SVC-6N4SJQIF3IX3FORG
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-6N4SJQIF3IX3FORG -m comment --comment "default/kubernetes:" -j KUBE-SEP-722QG7UQNTPWDFBY
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-WT6RQUWXRXGAUOJF
-A KUBE-SVC-LYRG26ZZO4GOQOI3 -m comment --comment "default/ng:" -j KUBE-SEP-7F6QLYX4EVXMHVGW
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-SFIQGU7OZTZRBGQ6I have fixed this issue adding a route:
ip route add 10.254.0.0/24 dev flannel.110.254.0.0/24 is my --service-cluster-ip-range. This is working but it sounds to me more luck than a real fix. Something else I can check, test or improve in my cluster?