I have a Kubernetes cluster on GKE that is configured to use a CE instance as an external NAT. However, I only want to route specific pods in the GKE cluster through the external NAT. Is this possible and how would I go about configuring this?
There isn't really an easy way to do with but it is possible. 1) You need to make sure to use VPC-Native 2) Make sure all your pods that will NAT are on the same node by taking advantage of advanced scheduling in k8s 3) Find the pod CIDR of that node using 'kubeclt describe no [node_name] | grep PodCIDR' 4) Create a custom route that sends all traffic from that CIDR through the NAT
This is not proven to work, to be honest. I know from dealing with some Cloud NAT issues that even using VPC Native pods, the pod IP is still sometimes going through SNAT on the node and thus takes the nodes IP. You could use the nodes internal IP instead of the pod CIDR on the node, this could possibly cause communication issues with the Master, but not necessarily.
Finally, keep in mind that this is not ideal since the node could end up being recreated and the IP or the pod CIDR may change.