By adding static route table on every node with proper rules, the container network also works fine. For example, given three nodes with three different docker bridge subnet:
node-1(192.168.0.1):
10.0.1.1/24
node-2(192.168.0.2):
10.0.2.1/24
node-3(192.168.0.3):
10.0.3.1/24On each node add the following routes:
ip route add 10.0.1.0/24 via 192.168.0.1 dev eth0
ip route add 10.0.2.0/24 via 192.168.0.2 dev eth0
ip route add 10.0.3.0/24 via 192.168.0.3 dev eth0With kube-proxy running in iptables mode, cluster-service-ip is translated to pod ip and finally routed to related node by the route table.
So what's the benefit of using cni plugin over route table? Is there a performance issue with route table method?
By design Kubernetes has a fluent structure. Pods, services, nodes can come and go depending on the needs, either by manual changes (rolling updates, new deployments) or by automatic scaling (HPA, node auto-scaling). Manually setting up rigid network structure negates the benefits of dynamic Kubernetes environment.
Overlay networks are not required by default, however, they help in specific situations. Like when we don’t have enough IP space, or network can’t handle the extra routes. Or maybe when we want some extra management features the overlays provide. One commonly seen case is when there’s a limit of how many routes the cloud provider route tables can handle. For example, AWS route tables support up to 50 routes without impacting network performance. So if we have more than 50 Kubernetes nodes, AWS route table won’t be enough. In such cases, using an overlay network helps.
It is essentially encapsulating a packet-in-packet which traverses the native network across nodes. You may not want to use an overlay network since it may cause some latency and complexity overhead due to encapsulation-decapsulation of all the packets. It’s often not needed, so we should use it only when we know why we need it.
https://itnext.io/an-illustrated-guide-to-kubernetes-networking-part-2-13fdc6c4e24c
If you are concerned with latency and overhead caused by CNI plugins here is a handy Benchmark results of Kubernetes network plugins