I have a Python script that can really eat some CPU and memory. Thus, I figure the most efficient way to run the script is to containerize it in a Docker Container. The script is not meant to run forever. Rather, it gets dependency information from environment variables, does it's behavior and then terminates. Once the script is over, by default Docker will remove the container from memory.
This is good. I am only paying for computing resource while the script is being run.
My problem is this: I have a number of different types of scripts I can run. What I want to do is create a manager that, given the name of a script type to run, gets the identified container to run in Google Container Engine in such as way that the invocation is configured to use a predefined CPU, disk and memory allocation envirnoment that is intended to run the script as fast as possible.
Then, once the script finishes, I want the container removed from the environment so that I am no longer paying for the resource. In other words I want to be able to do in an automated manner in Container Engine what I can do manually from my local machine at the command line.
I am trying to learn how to get Container Engine to support my need in an automated manner. It seems to me that using Kubernetes might be a bit of an overkill in that I do not really want to guarantee constant availability. Rather, I just want the container to run and die. If for some reason the script fails or terminated before success, the archtecture is designed to detect the unsuccesful attempt.
You could use a Kubernetes Controller to create a job object that 'runs to completion'.
A job object such as this can be used to run a single pod.
Once the job (in this case your script) has completed, the pod is terminated and will therefore no longer use any resources. The pod wouldn't be deleted (unless the job is deleted) but will remain in a terminated state. If required and configured correctly, no more pods will be created.
The job object can also be configured to start a new pod should the job fail for any reason should you require this functionality.
For more detailed information on this please see this page.
Also just to add, to keep your billing to a minimum, you could reduce the number of nodes in the cluster down to zero when you are not running the job, and then increase it to the required number when the jobs need to be executed. This could be done programmatically by making use of API calls if required. This should ensure your billing is kept as low as possible as you will only be billed for the nodes when they are running.