I have 2 clusters in GCP, one in Europe and the other in USA. I have created a VPC network to peer the subnetworks each others and configured the relative firewall rules. Now i'm able to create calls between pods, but i obtain a timeout when try to create them from a pod in europe to a service in the other cluster. I checked very well all firewall rules but can't find a solution. Can someone give me some hint to solve my problem?
The problem is that GCP requires to use a VM IP address in order to communicate outside the VPC. For allowing cross cluster communication on top of VPN, you need to make sure the clusters can communicate as they are on the same virtual network. GKE blocks egress traffic in case an internal IP address (pod address space) is used to access the internal IP address which are outside the virtual network (in this case through VPN). Hence you need to configure an ip table rule to masquerade traffic as it was originated from the VM instance IP address instead of the pod IP address for all outgoing traffic to the other subnet.
There is an implementation that is using daemon set to define the iptables MASQUERADE rules.
You can find more details on the github page - https://github.com/kubernetes-incubator/ip-masq-agent and on the k8s documentation - https://kubernetes.io/docs/tasks/administer-cluster/ip-masq-agent/