Pass date command as parameter to kubernetes cronjob

8/5/2019

I want to execute data migration through Kubernets cronjob using Luigi pipeline. My luigi task receives --start parameter which I want to pass through cronjob command.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: migration
spec:
  schedule: "0 0 */1 * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: spark
          containers:
          - name: migratecronjob
            image: latest-image
            imagePullPolicy: Always
            env:
              - name: TAG
                value: latest-image
              - name: MIGRATION_TAG
                value: migration-v05
            command:
              -  "luigi"
              - "--module" 
              - "module.task" 
              - "Migrate10Days"
              - "--start"
              - $(date +%Y-%m-%dT%H)
              - "--workers"
              - "10"
          restartPolicy: OnFailure

The cronjob cannot recognize the $(date +%Y-%m-%dT%H) as a bash script and pass this command as string to the luigi task.

-- Pedro Martins
cron
kubernetes
luigi

1 Answer

8/5/2019

I am not sure what are you going to archive, but this should work:

      - command:
        - sh
        - -c
        - 'exec luigi --module module.task Migrate10Days --start $(date +%Y-%m-%dT%H) --workers --workers'
-- FL3SH
Source: StackOverflow