I am trying to assign CPU resources for services running in kubernetes pods. Services are mostly nodejs based REST endpoints with some DB operations.
During the load test, tried different combinations between 100m and 1000m for pods. For the expected number of requests/second, when value is less than < 500m more pods are getting spawned as part of HPA than when the value is >500m. I am using CPU based trigger for HPA.
I couldn't figure out depending on what I shall be selecting particular CPU resource value. Can someone help me in this regard?
Two points:
If you configured the HPA to autoscale based on CPU utilisation, it makes sense that there are more replicas if the CPU request is 500m than if it's 1000m. This is because the target utilisation that you define for the HPA is relative to the CPU request of the Pods.
For example, if your target utilisation is 80% and the CPU request of the Pods is 500m, then the HPA scales your app up if the actual CPU usage exceeds 400m. On the other hand, if the CPU requests are 1000m, then the HPA only scales up if the CPU usage exceeds 800m.
Selecting resource requests (e.g. CPU) for a container is very important, but it's also an art in itself. The CPU is the minimum amount of CPU that your container needs for running reliably. What you could do to find out this value is running your app locally and try to evaluate how much CPU it is actually using, for example, with ps or top.