I'm playing around with Krustlet and MicroK8s and GKE.
If I create the Krustlet (node) in the same region as either a regional or zonal GKE cluster, when the Krustlet joins the cluster, the node reports that the RouteController created a route:
kubectl describe node/krustlet-01
Name: krustlet-01
...
Conditions:
Type Status Reason Message
---- ------ ------ -------
NetworkUnavailable False RouteCreated RouteController created a routeand:
gcloud compute routes list \
--filter=description="k8s-node-route" \
--format="table(destRange,nextHopInstance.scope())" \
--project=${PROJECT}
DEST_RANGE NEXT_HOP_INSTANCE
10.52.0.0/24 us-west1-a/instances/gke-clepsydra-default-pool-c8f03ba5-39v8
10.52.1.0/24 us-west1-c/instances/gke-clepsydra-default-pool-4c7699d9-dtq1
10.52.2.0/24 us-west1-b/instances/gke-clepsydra-default-pool-e515bbeb-v5hc
10.244.0.0/24 us-west1-a/instances/krustlet-01NOTE
krustlet-01is inus-west1-aand the cluster is inus-west1.
If I create Krustlets in other regions (us-west2|us-west3), the Krustlet fails with:
kubectl describe node/krustlet-03
Name: krustlet-03
...
Conditions:
Type Status Reason Message
---- ------ ------ -------
NetworkUnavailable True NoRouteCreated RouteController failed to create a route
Ready True KubeletReady kubelet is ready
OutOfDisk False KubeletHasSufficientDisk kubelet has sufficient disk space available
...
Events:
Reason Message
------ -------
FailedToCreateRoute Could not create route ... 10.244.0.0/24 for node krustlet-03 ... instance not foundNOTE edited for clarity
And, no routes are created even if I delete functioning Krustlet (which deletes its route).
The issue persists even if I try manually creating a route to one of the out-of-region Krustlets.
I create the cluster with:
gcloud beta container clusters create ${CLUSTER} \
--project=${PROJECT} \
--region=${REGION} \
--no-enable-basic-auth \
--release-channel "rapid" \
--machine-type "n1-standard-1" \
--image-type "COS_CONTAINERD" \
--preemptible \
--num-nodes="1Is this a permissions issue that, for some reason the cluster has insufficient permissions to create routes in different regions? If so, what's the remedy?
It would be understandable that, somehow the regional|zonal cluster is limiting traffic to the region but I don't see where this would be configured.
What am I not understanding?