I use airflow ( using Kubernetes Excutor ).
Setting :
parallelism = 50000
dag_concurrency = 30000
max_active_runs_per_dag = 20000
First, I created DAG containing 1000 tasks.
And trigger dag on webUI.
Then, dag was executed successfully and finished.
But,..
I created DAG containing 8000 over tasks.
And trigger dag on webUI.
Dag was executed successfully, but not running tasks (All task state is "None" and no progress.)
I have googled but I can’t find any (answer/solution).
Any ideas on how to solve this problem?
Any help will be appreciated.
Appreciate your help.