I have a multithreaded application which does a CPU-intensive task that I want to run on Kubernetes. The node I'm using has 56 cores and I set a request and limit of 2 cores for my pod.
Since it's CPU-intensive, typically there would be no point in having more threads than the # of cores (with hyperthreading, perhaps twice as many threads as cores), so I would allocate 2-4 threads and call it a day.
However, AFAIK, Kubernetes doesn't guarantee core affinity, so in the worst-case scenario, the 2 cores could be evenly split across the 56 cores, with each core working 2/56th of the time in parallel. If this happens and I allocate only 4 threads then at least 52 out of the 56 cores will be sitting idle.
If I understand it correctly, this problem is not unique to Kubernetes and applies to any virtualized environment where the hardware resources are shared.
What is the best practice when it comes to dealing with this potential worst-case scenario? Do you ignore it and assume you have high locality, or do you plan for the worst, or something in between?
Thread scheduling depends on OS including affinity, kubernetes doesn't affect it usually. If it's important to your workload - you can try Static policy in k8s to get exclusive CPU cores to your app.
You may use pod CPU limit 56 and run 56 threads to utilize all node cores. Assuming all other pods on this node use correct CPU requests, this should not affect other pods negatively.