I have a Kubernetes cluster and I have tested submitting 1,000 jobs at a time and the cluster has no problem handling this. I am interested in submitting 50,000 to 100,000 jobs and was wondering if the cluster would be able to handle this?
Yes you can but only if only don't run out of resources or you don't exceed this criteria regarding building large clusters.
Usually you want to limit your jobs in some way in order to better handle memory and CPU or to adjust it in any other way according to your needs.
So the best practice in your case would be to:
I hope you find this helpful.