I am writing an application that is running in my Google Kubernetes Engine(GKE) which is running on n2 machines (2 virtual CPU's per node). My cluster contains 3 nodes so 6 virtual CPUs.

I am running celery in my container with the following command:

celery -A myapp worker --loglevel=INFO --concurrency=10 -n worker1@%h

And I want to know what would be the optimal number of workers and optimal concurrency per worker in order to have the maximum amount of celery tasks running at the same time without affecting the performance or losing the tasks from redis.

Some explanation about the relationship between the two would be splendid. If I want to increase the number of tasks running, should I increase the number of workers, or concurrency of the same worker, or the GKE cluster size (number of nodes) to assign more CPUs?