Basically my cron is updating database every 2 minutes. But I need to run another cron which updates database every 24 hour. I need to make sure there are no race condition while the second cron (per day) is running i.e, both the cron can not run together and overlap. This needs to be implemented in Kubernetes cluster using jobs.
My solution is to run the first cron every 2 minutes but delay by 5 minutes when second cron runs (per day).
Any alternative solution is welcomed.
I tried to think about few possibilities but things can go wrong mainly by the fact that you have a job that is required to run every 2 minutes and let's suppose your daily job for any reason takes more time than you expect, a new 2 min job will start and this can't happen.
The best possible solution would be to solve this in a code level. You could create a lock in your application to prevent these jobs from running at the same time.
If you really can't come up with any solution to solve this problem in your code, there is a mechanism on Kubernetes that prevents a new job from the same cronjob to start. You can set the flag spec.concurrencyPolicy: Forbid so if your job takes longer than 2 minutes to start, the new job will be skipped. This is not a global flag, this is per cronjob only.
You can aggregate this possibility with spec.initContainers and find a way to run both tasks in the same cronjob.
spec.initContainers are specialized containers that run before app containers.
Here is an example:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: hello
spec:
concurrencyPolicy: Forbid
schedule: "*/2 * * * *"
jobTemplate:
spec:
template:
spec:
initContainers
- name: hello1
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
containers:
- name: hello2
image: busybox
args:
- /bin/sh
- -c
- DATE=`date +"%H:%M"`; if [[ $DATE == "00:00" ]]; then date; echo Hello from the Kubernetes cluster; fi
restartPolicy: OnFailure
This manifest is defining a cronJob that will run every 2 minutes and will start a initContainer (your 2 min job) and will also run a second container every 2 minutes. The thing is that in this second container we have a shell script checking if it's midnight (ignored the seconds as we can't be sure when it will start).
You may have to tune this part of the code:
DATE=`date +"%H:%M"; if [[ $DATE == "00:00" ]
Let's suppose your first container takes more than a minute to execute, it'll be 00:01 and your second job will never be executed.
Let me know if these information and solution helped you.