When running an autoscaling cluster on Google Cloud GKE, it is taking 15 minutes or sometimes half an hour with unschedulable pods before the autoscaler kicks in and provisions another node.
Especially in cases where I have manually deleted a node. The list of nodes shows as correct, but the number of nodes on the Node Cluster console will show as if the node was not deleted for at least 30 minutes.
Is there a way to force the autoscaler to take stock and make an upgrade immediately?
I have also tried turning off autoscaling and just setting a static number of nodes. But when deleting one of those nodes, it has not come back after waiting 45 minutes.
Is this expected behavior or is something up with GKE or do I potentially have something configured incorrectly?
I have checked and confirmed that I am not hitting up against any quotas. I have node auto-repair and autoscale both activated.
There is no way to force the autoscaler to go up (aside from manually changing the settings in the node pool). The autoscaler should scale up as soon as there are unschedulable pods as long as adding a new node will lead the pod to being scheduled.
Currently, only autoscaler actions are sent to Stackdriver so it's a lot harder to diagnose why an action was not taken. Your best bet would be to open a case with Google support if you have support or open an Issue Tracker to have a Googler check the logs for you.