We moved away from the Celery Executor in Airflow 1.10.0 because of some limitations of execution and right now we're using KubernetesExecutor
.
Right now we're not able to parallelize all the tasks in some DAGs even when we change the subdag_operator
in the code directly: https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/operators/subdag_operator.py#L38
Our expectations it's that with these modifications and using Kubernetes Executors we can fan out the execution of all tasks at the same time but we have the same behavior of the SequentialExecutor
.
This is the behavior that we have right now:
We would like to execute all of them at the same time using KubernetesExecutor
.