As host-gw use IP routes to subnets via remote machine IPs, it looks like pure L3 network solution.
Therefore, why need direct L2 connectivity between hosts?
host-gw adds route table entires on each host. And the entries are as following:
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.110.110.1 0.0.0.0 UG 100 0 0 eth0
10.100.14.0 10.110.110.21 255.255.255.0 UG 0 0 0 eth0
10.100.38.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
10.110.110.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
169.254.169.254 10.110.110.1 255.255.255.255 UGH 100 0 0 eth0The most import item is the value of Gateway(10.110.110.21). The route table will change the destination mac address to the mac_address of node(10.110.110.21) which is connected L2 directly to 10.110.110.22(current node).
If not L2 connected, the packet can not be delivered to nodes(next-hop)
host-gw adds route table entries on hosts, so that host know how to traffic container network packets.
This works on L2, because it only concerns hosts, switches and containers. switches does not care IP and route, hosts know containers exists, and how to route to them, containers just send and receive data.
If hosts are at different networks, L3 is introduced, and routers are involved. routers have no idea that containers exists, and any containers packet will be dropped, making communication impossible.
Of course, you can add route table entries on routers, but that is out of control flannel.