Need to scale nodes down instantly, Kubernetes doesn't allow developers to configure the time before a node scales down. Currently, it has an unconfigurable default of 10 minutes.
I'm developing an application which requires a lot of compute power/ hours. I'm looking at running process A, which adds some tasks to a job queue. Each of the jobs' require an average 10-40 mins for processing on a 6vCPU and 24GB instance.
When you apply kubectl scale deployment <deployment-name> it scales the pods down instantly. However, the underlying node will only scale down if no processing power has been used for 10 minutes.
Is there anyway to configure this wait-time? Is it worth it to contact Google GKE or Kubernetes to allow an exception for my particular use-case?
I also attempted to kill the node myself from within the pod after processing and was successful, however, Kubernetes recreates the deleted instance so my attempt was useless.
Any advice would be appreciated.
kubectl scale deployments mydeployment --replica=0
Actual: Scales pods down immediately but scales the underlying node down after 10 minutes. Requred: Scales pods and nodes down instantly.
It seems option like "-scale-down-unneeded-time_" is not supported. As workaround - If you're sure you're done with the node and no longer need it, you can scale down the GKE node pool manually, without waiting for Cluster Autoscaler to reclaim it automatically. A command to do that can be found here
However, this way you cannot control which node exactly will be turned down.
If you have a short-lived workload that occupies an entire VM and you want to turn down the VM as soon as it's done, you don't need Kubernetes, it will only get in the way. The best way to achieve this is to create a VMs on Compute Engine for every task, and turn each VM down as soon as it's done. Just make sure you put everything required to run the task in the VM image so you don't waste time downloading and installing stuff on startup.
Hope This help.