Let us assume that we have some predefined set of docker images on the private repository. Each of them takes a set of volumes, where input data are passed. Each image is a kind of a job - they represent an application that takes an input, computes the result, and then returns it as a file or something.
Currently, when the user wants to run a job, I can underneath invoke docker run -v ... [image] etc. Underneath, because everything is under some REST server API.
But, what if I want to schedule these images on the cluster? For simplicity let us assume that one image can be executed on a single machine, but in the future, I would like to have an option to schedule it to multiple machines.
So for example, now we have a cluster with machines A, B and C, user 1 invokes image x, and it is scheduled to machine A, then the same user schedules image y and it is scheduled to machine B (or A or C depending on their load), another user invokes image x with different input data (and thus volumes) and it is executed on machine A (or B or C - depending on their load; no dependencies between image and machine, or between images).
What can I use to achieve this? Can Kubernetes manage it? So far, looking at pods I think it cannot. Maybe just a load balancer is enough? Would like to take the most simple but also scalable and efficient solution.
A Kubernetes Job contains a pod template spec, so you just need to add to the PodSpec an anti-affinity rule with requiredDuringSchedulingRequiredDuringExecution where the exclusion selector will be the labels you set on the PodSpec.
Example:
apiVersion: batch/v1
kind: Job
metadata:
name: stackoverflow
spec:
template:
metadata:
labels:
foo: bar
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingRequiredDuringExecution:
- labelSelector:
matchExpressions:
- key: foo
operator: In
values:
- bar
topologyKey: "kubernetes.io/hostname"More details in the docs
This can also be achieved using docker swarm and a little logical scheduling. Here is a python docker scheduling library developed for a very similar use case.
It uses a resource list which defines the hosts in the swarm, a redis DB for logical resource management, and a static input mapping of where each expected image should find its input and write its output. Each "job" is then given a uuid which is passed to the image entrypoint and an output directory on the mount is created by the image entrypoint to write its unique output to. You can check out an example "job" in this context here where the entrypoint uses information from the job launcher.
The input and output volumes are mounted for the job and it is executed as a swarm service. Note that the scheduler can be configured to schedule in a few different ways, but by default should be SINGLE_NODE which should allow multiple jobs (up to the number of processes set in the resource list) to run on each host simultaneously.
It has been implemented as an async websocket service here, and a docker service running this is here.
Kube has some native ways of handling some similar batch scheduling, but I haven't explored those throughly yet. The DMOD project focuses on batch scheduling of distributed processing in a fully automated stack, but non-distributed batch scheduling is a consequence of that work.