RouteController "failed to create a route" between regions

5/13/2020

I'm playing around with Krustlet and MicroK8s and GKE.

If I create the Krustlet (node) in the same region as either a regional or zonal GKE cluster, when the Krustlet joins the cluster, the node reports that the RouteController created a route:

kubectl describe node/krustlet-01
Name:               krustlet-01
...
Conditions:
  Type                 Status    Reason                     Message
  ----                 ------    ------                     -------
  NetworkUnavailable   False     RouteCreated               RouteController created a route

and:

gcloud compute routes list \
--filter=description="k8s-node-route" \
--format="table(destRange,nextHopInstance.scope())" \
--project=${PROJECT} 
DEST_RANGE     NEXT_HOP_INSTANCE
10.52.0.0/24   us-west1-a/instances/gke-clepsydra-default-pool-c8f03ba5-39v8
10.52.1.0/24   us-west1-c/instances/gke-clepsydra-default-pool-4c7699d9-dtq1
10.52.2.0/24   us-west1-b/instances/gke-clepsydra-default-pool-e515bbeb-v5hc
10.244.0.0/24  us-west1-a/instances/krustlet-01

NOTE krustlet-01 is in us-west1-a and the cluster is in us-west1.

If I create Krustlets in other regions (us-west2|us-west3), the Krustlet fails with:

kubectl describe node/krustlet-03
Name:               krustlet-03
...
Conditions:
  Type                 Status  Reason                     Message
  ----                 ------  ------                     -------
  NetworkUnavailable   True    NoRouteCreated             RouteController failed to create a route
  Ready                True    KubeletReady               kubelet is ready
  OutOfDisk            False   KubeletHasSufficientDisk   kubelet has sufficient disk space available
...
Events:
Reason               Message
------               -------
FailedToCreateRoute  Could not create route ... 10.244.0.0/24 for node krustlet-03 ... instance not found

NOTE edited for clarity

And, no routes are created even if I delete functioning Krustlet (which deletes its route).

The issue persists even if I try manually creating a route to one of the out-of-region Krustlets.

I create the cluster with:

gcloud beta container clusters create ${CLUSTER} \
--project=${PROJECT} \
--region=${REGION} \
--no-enable-basic-auth \
--release-channel "rapid" \
--machine-type "n1-standard-1" \
--image-type "COS_CONTAINERD" \
--preemptible \
--num-nodes="1

Is this a permissions issue that, for some reason the cluster has insufficient permissions to create routes in different regions? If so, what's the remedy?

It would be understandable that, somehow the regional|zonal cluster is limiting traffic to the region but I don't see where this would be configured.

What am I not understanding?

-- DazWilkin
google-kubernetes-engine

0 Answers