airflow git-sync not refreshing dags when new dags are pushed to git repo

9/5/2019

I am using airflow on Kubernetes, with git-sync option for DAGS deployment. When there is a new DAGs pushed to our git repo, it airflow is not updating new DAGs until we restart scheduler pod. is there any other way to update the DAGs without restarting scheduler pod?

-- user2416
airflow
git
kubernetes

2 Answers

9/6/2019

I think you could use Jenkins.

With Jenkins it's easier to manage the automatic generation of dags,release them between environments and automatically menage versioning.

You wouldn't have to use scheduler then

enter image description here

-- jt97
Source: StackOverflow

10/4/2019

resolved this issue using git-sync sidecar container.

there is an extraContainers: option in values.yaml file. we can add kubernetes git-sync as a sidecar container to continuously sync from git.

extraContainers: 
  ## Additional containers to run alongside the Scheduler, Worker and Web pods
  ## This could, for example, be used to run a sidecar that syncs DAGs from object storage.
   - name: git-sync
     image: k8s.gcr.io/git-sync:v3.1.1
     volumeMounts:
      - name: ssh-key
        mountPath: /var/ssh
      - name: known-hosts
        mountPath: /var/knownhosts
      - name: git-sync-volume
        mountPath: /tmp/git

     env:
      - name: GIT_SYNC_REPO
        value: git@gitlab.example.com/test.git
      - name: GIT_SYNC_DEST
        value: git
      - name: GIT_SYNC_WAIT
        value: "10"
      - name: GIT_SYNC_SSH
        value: "true"
      - name: GIT_SSH_KEY_FILE
        value: /var/ssh/sshkey
      - name: GIT_SSH_KNOWN_HOSTS_FILE
        value: /var/knownhosts/known_hosts
  ## Additional volumeMounts to the main containers in the Scheduler, Worker and Web pods.
  extraVolumeMounts: 
     # - volumeMounts:
          - mountPath: "/var/ssh"
            name: ssh-key
            readOnly: true
          - mountPath: "/var/knownhosts"
            name: known-hosts
            readOnly: true
          - name: git-sync-volume
            mountPath: /tmp/git

  extraVolumes: 
     #- volumes:
        - name: ssh-key
          secret:
            secretName: dags-secret
            items:
            - key: key
              path: sshkey
        - name: known-hosts
          secret:
            secretName: dags-secret
            items:
            - key: known_hosts
              path: known_hosts

        - name: git-sync-volume
          emptyDir: {}
-- user2416
Source: StackOverflow