Airflow x Bitbucket (using Kubernetes Executor)

8/16/2019

I am trying to use Kubernetes Executor to process in separate pods the dags I created!

One of the steps we have is to sync git to access the dags and at first I put the following parameters in airflow.cfg but I can't process a simple python operator under these conditions (it doesn't create the pod). Has anyone managed to make this work? Are the settings for git sync correct?

Airflow.cfg:

# For either git sync or volume mounted DAGs, the worker will look in this subpath for DAGs
dags_volume_subpath =

# For DAGs mounted via a volume claim (mutually exclusive with git-sync and host path)
dags_volume_claim =

# For volume mounted logs, the worker will look in this subpath for logs
logs_volume_subpath =

# A shared volume claim for the logs
logs_volume_claim =

# For DAGs mounted via a hostPath volume (mutually exclusive with volume claim and git-sync)
# Useful in local environment, discouraged in production
dags_volume_host =

# A hostPath volume for the logs
# Useful in local environment, discouraged in production
logs_volume_host =

# A list of configMapsRefs to envFrom. If more than one configMap is
# specified, provide a comma separated list: configmap_a,configmap_b
env_from_configmap_ref =

# A list of secretRefs to envFrom. If more than one secret is
# specified, provide a comma separated list: secret_a,secret_b
env_from_secret_ref =

# Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
git_repo = git@bitbucket.org:myuser/airflow-kubernetes.git
git_branch = master
git_subpath = 

# Use git_user and git_password for user authentication or git_ssh_key_secret_name and git_ssh_key_secret_key
# for SSH authentication
git_user =
git_password =
git_sync_root = git@bitbucket.org:myuser/airflow-kubernetes.git
git_sync_dest = /airflow/dags

# Mount point of the volume if git-sync is being used.
# i.e. /airflow/dags
git_dags_folder_mount_point = /airflow/dags

enter image description here

enter image description here

-- Felipe FB
airflow
bitbucket
git
kubernetes

1 Answer

5/5/2020

I was able to get this working by configuring the following properties:

dags_in_image = False
dags_volume_subpath = dags
git_repo = https://github.com/<user-name>/<repo-name>.git
git_branch = master
git_subpath = dags
git_user = <git-username>
git_password = <git-password>
git_sync_root = /git
git_sync_dest = repo
git_dags_folder_mount_point = <path-to-dag-directory-locally>

My repo has one folder in it (dags) and the dags are stored in it.

-- prakshalj0512
Source: StackOverflow