We have several applications in a Kubernetes cluster that use Apache Ignite. Ignite creates various thread pools that are sized as follows:
Math.max(8, Runtime.getRuntime().availableProcessors())So basically the thread pool will always have at least size 8, but could be more if the system believes there are more processors.
The problem we're having is that some pods are spinning up with pool size 8, and others are using size 36, which is the number of CPUs on the node.
We're using Helm to deploy all apps, but we're not setting any CPU limits for any pods. In theory they should all see the same number of available CPUs.
What else could cause pods on the same node to see different views of how many processors are available?
We have a health end point in all of our apps that shows the number of CPUS reported by the JVM, using the same Runtime#availableProcessors() method that Ignite uses.
All of our apps, including ones where Ignite thinks there are 36 CPUs, report 2 processors once the process has started.
I found this interesting line in the Java documentation for that method:
This value may change during a particular invocation of the virtual machine. Applications that are sensitive to the number of available processors should therefore occasionally poll this property and adjust their resource usage appropriately.
It seems we're in a race condition where early on in app startup, that value reports 36 but at some point drops to 2. Depending on when the Ignite beans are fired, they're seeing either 36 or 2.
Very strange indeed. Are they in the same namespace? Is there any quota or Limit range the helm chart or you have set?
Also, can you verify the limits of cpu by running kubectl get nodes -o custom-columns='NAME:metadata.name,CPU:status.capacity.cpu' ?
tl;dr The underlying issue seems to be when resources.requests.cpu is set exactly to 1000m.
I wrote a simple Java app that dumps the available number of processors:
public class CpuTest {
public static void main(String[] args) {
System.out.println("Number of CPUs = " + Runtime.getRuntime().availableProcessors());
}
}
I packaged into a Dockerfile and created a simple deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: cputest
labels:
app: cputest
spec:
replicas: 1
selector:
matchLabels:
app: cputest
template:
metadata:
labels:
app: cputest
spec:
containers:
- name: cputest
image: dev/cputest:latest
imagePullPolicy: NeverI ran this on my local RedHat 7 machine which has 24 cores. The output was expected:
Number of CPUs = 24I then applied various CPU resource requests to the deployment:
resources:
requests:
cpu: 1000mand re-deployed. The results were interesting:
So the issue only arises when the CPU request is set 1000m (also tried 1 and got the same result, where it thinks it has all 24 CPUs).
I went back and looked at all of our apps. Sure enough, the ones where we set the CPU request to exactly 1000m are the ones that have the issue. Any other value works as expected.
Of inerest, when I also set CPU limit to 1000m, the problem goes away and the JVM reports 1 CPU.
It very well could be this is expected and I don't fully understand how CPU resource and limits are used by Kubernetes, or perhaps an issue with the version we're on (1.12.7).
Either way I at least have an answer as to why some of our pods are seeing different CPUs.