We have a couple of clusters running on GKE and up until now I've only been maintaining a CPU request/limit for pods. We've recently run into issues where the cluster autoscaling isn't responding when pods begin to be evicted for low memory, and we can visibly see in the GKE console that there is memory pressure on at least one of the nodes.
I was hoping someone could tell me: is there some sort of calculation that we can make as a starting point for how much memory we should request/limit per pod of each of our services, or is that was more trial/error? Is there some statistic service that can track what's being used in the cluster now?
Thanks!
There is no magic trick for calculating limits. You need to start with reasonable limits and refine using trial and error.
I can suggest a video from YouTube that explains quite well a method to refine your limits: https://youtu.be/-lsJyni7EQA
Basically it suggests to start with low limits and load test your application (one pod instance) until it breaks. Than, raise the limits and load test again until you find good values.