There is an option of mounting volumeMount for dags and logs under kubernetes section of airflow.conf e.g.
[kubernetes]
airflow_configmap = airflow_config
worker_container_repository = airflow
worker_container_tag = runner2
worker_container_image_pull_policy = IfNotPresent
delete_worker_pods = true
dags_volume_claim = airflow-dags-pvc
dags_volume_subpath = airflow/development/dags
logs_volume_claim = airflow-logs-pvc
logs_volume_subpath = airflow/development/logs
namespace = developmentThis works as expected. Means I could see worker pods successfully mounting both of these volumes and its relevant volumeMounts inside the container.
"volumeMounts": [
{
"name": "airflow-dags",
"readOnly": true,
"mountPath": "/usr/local/airflow/dags",
"subPath": "airflow/development/dags"
},
{
"name": "airflow-logs",
"mountPath": "/usr/local/airflow/logs",
"subPath": "airflow/development/logs"
},
BUT, My worker pods have dependency of picking up custom airflow plugins from directory airflow/development/plugins and airflow/development/libs. Due to which I need to add more volumeMount into the worker pod with relevant subPaths from NFS server. How can I achieve that ? I tried searching for any relevant config value but couldn't find any.
Update: I was passing executor_config into the one of the dags sensors task as executor_config={"KubernetesExecutor": {"image": "airflow:runner2"}}. By the look at code airflow/contrib/kubernetes/worker_configuration.py , it seems like if I pass in volumes and volumeMounts as part of this configs, it should work. I will try this and update here.
Sorry i'd to answer my own question. Maybe will help someone. Inside the dag file, where I define the task, I simply had to add the executor_config as follows;
IngestionStatusSensor(
task_id=...,
executor_config={"KubernetesExecutor": {
"image": "airflow:runner2",
"volume_mounts": [
{
"name": "airflow-dags",
"mountPath": "/usr/local/airflow/libs",
"subPath": "airflow/development/libs"
},
{
"name": "airflow-dags",
"mountPath": "/usr/local/airflow/plugins",
"subPath": "airflow/development/plugins"
}],
}
},
dag=dag,
ingestion_feed=table,
poke_interval=60 * 30,
offset=0,
start_date=start_date
)
where airflow-dags volume is already defined by pod creator that claims the PVC defined in configs under kubernetes section of airflow.conf e.g. dags_volume_claim = airflow-pvc.