Kubernetes executor do not parallelize sub DAGs execution in Airflow

9/3/2018

We moved away from the Celery Executor in Airflow 1.10.0 because of some limitations of execution and right now we're using KubernetesExecutor.

Right now we're not able to parallelize all the tasks in some DAGs even when we change the subdag_operator in the code directly: https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/operators/subdag_operator.py#L38

Our expectations it's that with these modifications and using Kubernetes Executors we can fan out the execution of all tasks at the same time but we have the same behavior of the SequentialExecutor.

This is the behavior that we have right now:

enter image description here

We would like to execute all of them at the same time using KubernetesExecutor.

-- Flavio
airflow
airflow-scheduler
kubernetes

0 Answers