I have a node pool with a minimum pool size of 0, and a max pool size of 3. Normally, nothing is happening on this node pool, so GKE correctly scales down to zero. However, if I tried to submit a job to this pool via kubectl, the pod fails with Unschedulable.
I can run a kubectl with --enable-autoscaling --min-nodes 1 --max-nodes 3 wait 10 seconds, and then deploy, and then wait for completion before changing the min-nodes back to 0 but this doesn't seem ideal.
Is there a better way to get the pool to start a node when a pod is pending?
Even with something like taints or nodeAffinity, I don't think you can tell Kubernetes to spin up nodes in order to schedule workloads. The scheduler requires a node to be available already.
(Out of curiosity, how were you scheduling jobs to a specific nodepool via kubectl?)
As the autoscaler is based on Pod resource requests to scale up/down, you need at least 1 value that autoscaler can use as base calculation to know if the pool need additional node or not.
Here is more information about How cluster autoscaler works [1]