I have setup an airflow instance on eks and I set AIRFLOW__CORE__EXECUTOR=KubernetesExecutor
. When a dag has only tasks but not a subdag then each pod is dynamically created for each tasks and runs fine but when I have a dag with multiple subdags, it does not runs the subdag and instead the first task inside the first subdag remains in the queue state.
After I set AIRFLOW__CORE__EXECUTOR=LocalExecutor
and while creating the subdag I set executor=KubernetesExecutor()
it works fine and tasks inside the subdags runs in separate pods. But I don't know whether I will get the same power of the KubernetesExecutor
or not.
Thanks in advance.