I have a K8S cluster with 20 Nodes, on AWS. I need to make a pipelines in a way such that a new Pod will be deployed only on one dedicated Node and once its execution has reached a particular stage, it will be deleted from that node, making it idle so that another dedicated node can be deployed on that node.
I have tried using nodeName, which is fine but we need to make scheduling automatic. In nodeName method, I had to see if any node is idle and then deploy it there. Also, I need to check if the execution is done for every node and remove the pods if its done. This is also done manually. Is there any way to automate this as well?
I'm pretty sure the cluster-autoscaler will do what you describe. You'd want to create an Auto Scaling Group with two Tags:
k8s.io/cluster-autoscaler/node-template/taint/my-special-key and a value of whatever:NoSchedulek8s.io/cluster-autoscaler/node-template/label/my-special-label and a value of whatever-you-want (or even a blank value)then when you create the Pod, put its requests: with exactly the size of memory: and cpu: to fit the Node, include a nodeSelector of my-special-label: whatever-you-want (or ""), and a toleration of the taint
Then, the cluster-autoscaler will create a new Node because it knows about the label and taint limitation, and will delete the Node after 10 minutes (which I'm pretty sure is customizable) if a Pod isn't using it.
No other Pod will trigger that scaling event and no other Pod will be scheduled onto the Node