We have Openshift cluster 3.11, where we have configured Horizontal Autoscaler:
Min Pods: 5
Max Pods:10
Cpu Request Target: 80%1) We have a spring application running in pods which persist data to mongodb. I have observed that when processing starts, it does increase pod count. However it also starts terminating pods when CPU utilization is less.
I have observed that when pods are terminated, it does not consider any processing running on that pod. Due to the same, i do find few database records are lost(records which were about to be saved from the pod which got terminated): mongodb collection count.
2) When i ran with fixed number of pods, i do not loose data.
Is there any way to tune Autoscaler so that it does not terminate pod , if there is any processing running ( for e.g async database save operations) on the pod
Or Terminate the pod only after particular duration even though the CPU utilization less?
(quote) "...Or Terminate the pod only after particular duration even though the CPU utilization less?" - take a look at terminationGracePeriodSeconds in PodSpec.
If you are in control of the application and assuming a shell script kicks off the application, there may be some processing you can do in the shell script in response to SIGTERM (TERM signal) to handle graceful termination. See also Termination of Pods.