Allocating temporary VMs to parallelize one-shot batch jobs (GCP, Azure or AWS)

6/20/2017

I am evaluating options for launching arbitrary Python tasks/scripts on temporary cloud VMs that get shut down once the job is complete. I am looking across all cloud providers, but the ideal solution should not be vendor-specific. Here is what I found:

  • Docker Swarm / Kubernetes / Nomad for spinning up docker containers. All look attractive, but cannot confirm if they can terminate VMs once the task is complete.
  • Cloud Functions/Lambdas look great, but works only for short-lived (few minutes) tasks. Also, GCP supports only JavaScript.
  • Spins up/down VMs explicitly from a launch script with vendor specific commands. Straightforward and should work.
  • AWS Batch, Azure Batch - vendor-specific services for batch jobs
  • AWS Data Pipeline, Azure Data Factory, Google Dataflow - vendor-specific services for data pipelines

Did I miss any good option? Does any container orchestration service like Docker Swarm support allocation and deallocation of multiple temporary VMs to run a one-shot job?

-- J. S.
amazon-web-services
azure
docker-swarm
google-cloud-platform
kubernetes

0 Answers