Kubernetes can jobs be chained together as steps in a workflow?

9/13/2018

Reading the Kubernetes "Run to Completion" documentation, it says that jobs can be run in parallel, but is it possible to chain together a series of jobs that should be run in sequential order (parallel and/or non-parallel).

https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/


Or is it up to the user to keep track of which jobs have finished and triggering the next job using a PubSub messaging service?

-- HashRocketSyntax
data-science
jobs
kubernetes
pipeline
workflow

2 Answers

9/13/2018

I have used initContainers under the PodSpec in the past to solve problems like this: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox
    command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
  - name: init-mydb
    image: busybox
    command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']

Take a look here for the chaining of containers using the "depends" keyword is also an option:

https://github.com/kubernetes/kubernetes/issues/1996

-- Mike
Source: StackOverflow

9/14/2018

Overall, no. Check out things like Airflow for this. Job objects give you a pretty simple way to run a container until it completes, that's about it. The parallelism is in that you can run multiple copies, it's not a full workflow management system :)

-- coderanger
Source: StackOverflow