Blocking execution of a k8s job till another one gets completed in a helm chart

9/24/2020

I have a single helm chart and there are two job resources defined under the template.

.
├── Chart.yaml
├── templates
│   ├── job.yaml
│   └── post-copy-job.yaml
└── values.yaml

My first job (job.yaml) is actually a single template but it rolls out multiple jobs (using range).

I want my second job template (post-copy-job.yaml) to hold execution until all the jobs under the first template get completed. I tried to use post-install hook in the second job template.

annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation

Still, it gets executed and completed before all the jobs under job template one gets completed. My question is how can I delay the execution of the second job template till all the resources under the job template one get completed.

-- Rajiv Rai
kubernetes
kubernetes-helm

1 Answer

9/27/2020

While David comment is probably the best way to go, I think the most lightweight solution you can go with is deploying another job that will be stuck in a loop and constantly monitor the status of the first jobs, and once they all finish it will start processing. This will consume a bit of extra resources but will get you out of your pickle with minimal effort.

You can monitor the first job by either using the k8s API, and if you want to cut corners there as well you can use a blob storage for that - creating a blob for each job and marking it as done when the job finishes.

-- Yaron Idan
Source: StackOverflow