I have multiple Kubernetes clusters. they are all on the "same network" - that is, there are other servers/services on the network such that all pods in all clusters can access those servers/services.
I know that each pod gets a unique IP and thus all pods can communicate with each other directly with those IPs. BUT (AND HERE IS MY QUESTION) - do pods get unique IPs across my whole network? or just across/within the cluster?
Reason for this question is - can I have all the clusters use the same network range for pods. that is, is this proper design:
DESIGN A
CLUSTER 1
Pod1.1 10.0.0.1
Pod1.1 10.0.0.2
...
Pod1.1 10.0.0.n
CLUSTER 2
Pod2.1 10.0.0.1
Pod2.1 10.0.0.2
...
Pod2.1 10.0.0.nor, do I need to have it like this:
DESIGN B
CLUSTER 1
Pod1.1 10.0.0.1
Pod1.1 10.0.0.2
...
Pod1.1 10.0.0.n
CLUSTER 2
Pod2.1 10.0.1.1
Pod2.1 10.0.1.2
...
Pod2.1 10.0.1.nANOTHER WAY OF ASKING THIS QUESTION:
When a Pod1.1 calls a webserver outside of the cluster, does it advertise it self as 10.0.0.1? or as some NAT IP that kubernetes handles and maps back to 10.0.0.1?
Thanks!
do pods get unique IPs across my whole network? or just across/within the cluster?
Generally, just within the cluster. When you set up your network overlay you set it up with a PodCidr and that will be the pod network within your cluster. You could have two clusters with the same PodCidr and still able to communicate with each other. Once traffic leaves the cluster and a server for that matter it's seen with the external IP.
So yes, there is some trickery with iptables (or depending the overlay) to make it look like it's 'local' within the cluster, but outside of the cluster, it's just IP routing.
Having said that, some network overlays allow you to connect two or more clusters with each other. For example, Calico BGP Peering and Cillium Multi-cluster.