I'm building a Kubernetes cluster with Flannel as my CNI. (on-premise hardware cluster running Centos7) I ran into a problem, and I'd like to know if this is a misconfiguration on my part, or if it's just the way Flannel works.
My OS's all sit on a 10.~/24 network. I've provisioned Kubernetes to use a 172.~/16 so that it would have room to provision all of its pods and nodes and containers, oh, my! When I attempt to access external services like an HDFS cluster, the pods send packets out, but never receive a reply.
The problem I discovered is that MySQL is receiving packets with an IP address in the 172.~/16 range, and doesn't know where to send them. I can set up my network to send everything in that range to a specific K8s server, but then it locks down which server the pod can run on. That's kind of ruining the purpose of k8s.
I was expecting flannel to masquerade packets on their way out, so that external servers know how to route their responses. Is this just the way Flannel is designed, or did I miss a step somewhere?
You can use SNAT to masquerade the pod IP into the node address when sending outbound traffic so it can go back to its original destination.
There is a Masquerade Agent that you can deploy in your cluster to determine what CIDRs are to be masked for outbound traffic.
Additionally, Flannel has built-in masquerading option that you can use if you don't want to rely on the aforementioned agent.