I am using airflow on Kubernetes, with git-sync option for DAGS deployment. When there is a new DAGs pushed to our git repo, it airflow is not updating new DAGs until we restart scheduler pod. is there any other way to update the DAGs without restarting scheduler pod?
I think you could use Jenkins.
With Jenkins it's easier to manage the automatic generation of dags,release them between environments and automatically menage versioning.
You wouldn't have to use scheduler then
resolved this issue using git-sync sidecar container.
there is an extraContainers:
option in values.yaml file. we can add kubernetes git-sync as a sidecar container to continuously sync from git.
extraContainers:
## Additional containers to run alongside the Scheduler, Worker and Web pods
## This could, for example, be used to run a sidecar that syncs DAGs from object storage.
- name: git-sync
image: k8s.gcr.io/git-sync:v3.1.1
volumeMounts:
- name: ssh-key
mountPath: /var/ssh
- name: known-hosts
mountPath: /var/knownhosts
- name: git-sync-volume
mountPath: /tmp/git
env:
- name: GIT_SYNC_REPO
value: git@gitlab.example.com/test.git
- name: GIT_SYNC_DEST
value: git
- name: GIT_SYNC_WAIT
value: "10"
- name: GIT_SYNC_SSH
value: "true"
- name: GIT_SSH_KEY_FILE
value: /var/ssh/sshkey
- name: GIT_SSH_KNOWN_HOSTS_FILE
value: /var/knownhosts/known_hosts
## Additional volumeMounts to the main containers in the Scheduler, Worker and Web pods.
extraVolumeMounts:
# - volumeMounts:
- mountPath: "/var/ssh"
name: ssh-key
readOnly: true
- mountPath: "/var/knownhosts"
name: known-hosts
readOnly: true
- name: git-sync-volume
mountPath: /tmp/git
extraVolumes:
#- volumes:
- name: ssh-key
secret:
secretName: dags-secret
items:
- key: key
path: sshkey
- name: known-hosts
secret:
secretName: dags-secret
items:
- key: known_hosts
path: known_hosts
- name: git-sync-volume
emptyDir: {}