My understanding is that in Kubernetes, when using the Horizontal Pod Autoscaler, if the targetCPUUtilizationPercentage field is set to 50%, and the average CPU utilization across all the pod's replicas is above that value, the HPA will create more replicas. Once the average CPU drops below 50% for some time, it will lower the number of replicas.
Here is the part that I am not sure about:
What if the CPU utilization on a pod is 10%, not 0%?Will HPA still terminate the replica?
10% CPU isn't much, but since it's not 0%, some task is currently running on that pod. If it's a long lasting task (several seconds) and HPA decides to terminate the pod, that task will not be finished.
Does the HPA terminate pods only if the CPU utilization on them is 0% or does it terminate them whenever it sees that the value is below targetCPUUtilizationPercentage?
How does HPA decide which pods to remove?
Thank you!
So you have two questions in there and let me address one by one. The first part - if a pod in a replica set is consuming let's say 10% then will Kubernetes kill that pod? The answer is Yes. Kubernetes is not looking at the individual pods but at an average of that metric across all pods in that replica set. Also the scaling down is gradual as explained here
The second part of the question - how does your application behave gracefully when a pod is about to be killed and it is still serving some requests? This can be handled by the grace period of the pod termination and even better if you implement a PreStop hook - which will allow you to do something like stop taking incoming requests but process existing requests. The implementation of this will vary based on the language runtime you are using, so I won't go in the details here.
Lastly - one scenario you should consider is what if VM on which pod was running goes down abruptly - you have no chance to execute PreStop hook! I think the application needs to be robust enough to handle failures.