I am running Airflow with CeleryExecutor and they are package and deployed with helm charts. I have one worker that I can scale up/down by specifying the number of replicas in the yaml file. This worker is running as StatefulSet and by default it uses the airflow queue. Is it possible to create another StatefulSet chart for another worker and specify the queue name? This way for the DAG definition files, I can specify the queue to use as well.
In your use case it would be better to use Kubernetes Executor instead of splitting the worker management logic between StatefulSets.
The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance.
You can find more details about it in this documentation.
And also I recommend this blog.
I hope it helps.