I am implementing a cluster-api controller using Kubernetes as the infrastructure provider - that is, I am trying to run Kubernetes Nodes as Kubernetes Pods and form a cluster within a cluster.

I have this working apart from network connectivity between Pods of the inner cluster (running on Pods of the infrastructure cluster), but I am stuck as to what the issue is.

I am running on GKE, using their default CNI implementation. I am then attempting to use Calico for an overlay implementation of the inner cluster, using IP-in-IP encapsulation so the Nodes of the infrastructure cluster do not need to know how to route inner cluster Pod IPs.

I am creating the infrastructure cluster as follows (the UBUNTU image is needed for the ipip kernel module required by Calico's IP-in-IP encapsulation.

gcloud container clusters create management-cluster --image-type=UBUNTU

I then deploy a number of nginx Pods to the inner cluster. If they land on the same inner cluster Node, they can connect to eachother. If they land on separate inner cluster Nodes they cannot, so I assume this means the IP-in-IP tunnel isn't working properly, but I am not sure why. This fails even if the inner cluster Nodes (Pods) land on the same infrastructure (outer cluster) Node. Pod and Service CIDR ranges of the two clusters do not overlap.

I realise this is not a supported use case for Calico, but I cannot see a reason why it is not possible and would like to get it working. Do the outer cluster Nodes need to support forwarding IP-in-IP packets? They are configured to forward IPv4 packets, but I am not sure if that is enough.

I guess more information is required to give a concrete reason for why this isn't working, but I am not too sure what that would be at this point and would be grateful for any direction.