I am trying to run a job on GKE for 5 mins and 50 nodes. However when i scale down instances it happens sequentially and thus costing me much more for a 4-5 min job.
Is there any way to paralelly delete GKE instances?
If you are using already to scale down your cluster:
$ gcloud container clusters resize [CLUSTER_NAME] --node-pool [NODE_POOL] --size [SIZE]I believe there are other option to speed up this process. On other other hand if you are reeling on autoscaler you could try if with that command the resize is faster and you would be able to meet your requirements. However keep in mind that the purpose of Kubernetes is not create an infrastructure were it is extremely fast to spin up and add 50 nodes and when you are done to kill them.
Consider also from the doumentation:
The kubectl drain command should only be issued to a single node at a time. However, you can run multiple kubectl drain commands for different node in parallel, in different terminals or in the background. Multiple drain commands running concurrently will still respect the PodDisruptionBudget you specify.
Therefore it turns out from the documentation that drain a node (that is an essential phase to remove a node) it is discouraged
Kubernetes cluster has an underlying Instance Group.
I was able to delete the nodes in parallel by directly changing the number of nodes in Instance Group from 50 to 5.
All nodes were deleted within 30 seconds and GKE had also automatically updated the cluster size with the new value.