In k8s environment build with virtualbox, nodes in flannel network can't ping each other

6/14/2017

Env info: I build my k8s cluster with virtualbox on mac.The node os is centos7.3. There are two node and one master, all of which's network is Net(can visit the public network) and Host-Only (can visit the inner network).The IP info is following:
master:
network enp0s3 :192.168.99.100/24 (Host-only network,node1 and node2 can visit this IP)
network enp0s8 :10.0.3.15/24 (Net network)
node1:
network enp0s3 :192.168.57.3/24 (Host-only network,master and node1 can visit this IP)
network enp0s8 :10.0.3.16/24(Net network)
node2:
network enp0s3 :192.168.58.2/24(Host-only network, master and node1 can visit this IP)
network enp0s8 :10.0.3.17/24(Net network)

k8s version is:
kubernetes(v1.5.2),ectd( 3.1.7),flannel(0.7.0)。

Network set on master:
etcdctl set /atomic.io/network/config '{"Network":"172.17.0.0/16"}'

flannel set on node1:
/run/flannel/subnet.env

FLANNEL_NETWORK=172.17.0.0/16
FLANNEL_SUBNET=172.17.94.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=true

/etc/sysconfig/flanneld:

FLANNEL_ETCD_ENDPOINTS="http://192.168.99.100:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_OPTIONS="-iface=enp0s3 -public-ip=192.168.57.3 -ip-masq=true"

flannel set on node2: /run/flannel/subnet.env :

FLANNEL_NETWORK=172.17.0.0/16
FLANNEL_SUBNET=172.17.50.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=true

/etc/sysconfig/flanneld:

FLANNEL_ETCD_ENDPOINTS="http://192.168.99.100:2379"
FLANNEL_ETCD_PREFIX="/atomic.io/network"
FLANNEL_OPTIONS="-iface=enp0s3 -public-ip=192.168.58.2 -ip-masq=true"

node1's route:

flannel0:172.17.94.0/16
docker0:172.17.94.1/24

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.3.2 0.0.0.0 UG 100 0 0 enp0s8
10.0.3.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s8
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel0
172.17.94.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.57.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s3

and

default via 10.0.3.2 dev enp0s8 proto static metric 100 
10.0.3.0/24 dev enp0s8 proto kernel scope link src 10.0.3.16 metric 100 
172.17.0.0/16 dev flannel0 proto kernel scope link src 172.17.94.0 
172.17.94.0/24 dev docker0 proto kernel scope link src 172.17.94.1 
192.168.57.0/24 dev enp0s3 proto kernel scope link src 192.168.57.3 metric 100 

node2's route:
flannel0: 172.17.50.0/16
docker0: 172.17.50.1/24

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.3.2 0.0.0.0 UG 0 0 0 enp0s8
10.0.3.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s8
169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 enp0s8
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 flannel0
172.17.0.0 0.0.0.0 255.255.0.0 U 1 0 0 flannel0
172.17.50.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
192.168.58.0 0.0.0.0 255.255.255.0 U 100 0 0 enp0s3

and

default via 10.0.3.2 dev enp0s8 
10.0.3.0/24 dev enp0s8 proto kernel scope link src 10.0.3.17 
169.254.0.0/16 dev enp0s8 scope link metric 1003 
172.17.0.0/16 dev flannel0 
172.17.0.0/16 dev flannel0 scope link metric 1 
172.17.50.0/24 dev docker0 proto kernel scope link src 172.17.50.1 
192.168.58.0/24 dev enp0s3 proto kernel scope link src 192.168.58.2 metric 100 

Then, Ping node2's docker ip(172.17.50.1 ) on node1 is not ok, Ping node1's docker ip(172.17.94.1) is not ok either. Use tcpdump to see the network, found maybe the network is not config ok.The source ip and des ip should be 192.168.57.3(through enp0s3) but not 10.0.3.16(through enp0s8): enter image description here

I don't know why the nodes can't visit througt flannel.Hoping for help, thanks.

-- liyj144
flannel
kubernetes
virtualbox

0 Answers