I am going through kubernetes the hard way to learn more about how to use it. I am implementing it on openstack vm's. The pod networks local to each individual kubernetes worker need to be able to communicate with one another via static routes. In order to set up static routes with IP addresses other than the primary IP address configured in openstack I have to use allowed_address_pairs in openstack. This works if I try to route on eth0, but if I try to route on the CNI interfaces (cnio0) it fails.

I configured static routes as a test between secondary IP addresses configured as aliases to the worker nodes' primary interfaces. I set ip forwarding to 1 in proc. In openstack, I configured the allowed_address_pairs using the secondary IP addresses and their associated mac addresses.

This routing configuration worked.

I then enabled the cni and fired up a few pods which triggered kubelet to create a cnio0 interface on each host and apply the POD CIDR IPs to those interfaces. I added these IPs and their respective mac addresses to the allowed_address_pairs as well, but these interfaces do not route. I am not able to reach the IPs from each individual box.

I expected to be able to successfully ping from node 1's cni0 IP address to node 2's cni0 IP address.

I watched the communication between the hosts in tcpdump while pinging from node 1's cnio0 interface to node 2's cnio0 interface. On node 1 I can see traffic going out. On node 2 I can see the traffic being received and sent back, but I never see the return packet on node 1.

This is the exact same behavior I saw on the aliased interfaces when I tested static routes between them without configuring the allowed_address_pairs in openstack.

How can I get the cnio0 interfaces to work with allowed_address_pairs so they can route between one another?