Airflow: Dag containing many tasks is able to be running, but not progress from task status = 'None'

12/19/2019

I use airflow ( using Kubernetes Excutor ).

Setting :

parallelism = 50000
dag_concurrency = 30000
max_active_runs_per_dag = 20000

First, I created DAG containing 1000 tasks.

And trigger dag on webUI.
Then, dag was executed successfully and finished.
But,..
I created DAG containing 8000 over tasks.
And trigger dag on webUI.
Dag was executed successfully, but not running tasks (All task state is "None" and no progress.)

Airflow UI

I have googled but I can’t find any (answer/solution).
Any ideas on how to solve this problem?
Any help will be appreciated.
Appreciate your help.

-- 1220122
airflow
airflow-scheduler
kubernetes

0 Answers