My goal

I am trying to understand and figure out best practices or optimal ways or best methods how to properly set up resources for low performance-hungry and high performance-hungry Java Springboot app.

Examples

First example: Let have low performance-hungry Springboot App which only computes a key for cache and calls Redis for data with the key. I have tried two configurations of resources and java opts.

replicaCount: 4
memory: 1.9Gi
cpuRequest: 800m
cpuLimit: 1200m
JAVA_OPTS: "-XX:+UseG1GC -XX:MaxRAM=1792m -Xmx768m -Xms768m -XX:MaxMetaspaceSize=1024m -XshowSettings:vm -XX:ActiveProcessorCount=2"

Performance is good and the application has a response time around 3.5 ms in the median. And for 0.99 percentile it is 90ms. GC pause count was 0.4 per minute and pause duration 20ms. Also little throttling.

Then I have tried this setup:

replicaCount: 4
memory: 3Gi
cpuRequest: 800m
cpuLimit: 10000m
JAVA_OPTS: "-XX:+UseG1GC -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=80 -XshowSettings:vm"

The application was more memory-hungry but the response time was the same, 3.5ms in the median. But only on 0.99 percentile was 72 ms. GC pause count was lower, something about 0.1 per minute, and pause duration 5 ms. Throttling during startup but no throttling during the run.

Second example: Performance hungry Springboot application which loads data from DB calculates multiple things like the distance between two points, calculation of price for multiple deliveries, filtering possible transport or pickup points for the package.

It was running on 4 VPS with 4 CPUs and 4 GB. But on Kubernetes, it needs more resources than I expected.

replicaCount: 4
memory: 7.5Gi
cpuRequest: 2000m
cpuLimit: 50000m
JAVA_OPTS: "-XX:+UseG1GC -XX:MaxRAMPercentage=80 -XX:InitialRAMPercentage=80 -XX:+UseStringDeduplication -XshowSettings:vm"

Performance is good, but vertical scaling was doing nothing, but only horizontal scaling provided better performance. CPU usage reported by Kubernetes is about 1 and has no throttling.

What I have done so far

• Read multiple articles found via Google, but any not give me a proper answer or explanation.
• I have tried various settings for CPU and memory resource limits on Kubernetes spec, but it was not doing what I expected. I expected, which is lowering response time and the ability to process more requests.
• Scaling vertically does not help either, everything was still slow.
• Scaled horizontally pods with low CPU and memory with specified Xms, Xmx, ... it does that, pods were stable, but not performant as possible. (I think.) And also some throttling of CPU also if CPU was not fully used.

Question

I have a big problem with properly setting memory and CPU on Kubernetes. I do not understand why memory usage is increasing when I give it more CPU (Xss is the default value) and usage is the same. The pod is not OOM killed only if a gap between committed memory and used memory is about 1 GB (for the second application example) or 500MB (for the first application example).

If I set Xmx, Xms, MetspaceSite, and Xss, then I can achieve lower memory usage and CPU usage. But in this case, increasing the pod memory limit is complicated because it is not defined as a percentage and I must every time calculate each Java opt.

Also If give application too much memory, at the begging it will start at some level, but after some time it every times goes to limit (until gap between committed and heap memory is 1GB - 500MB) and stays there.

So, how is the proper way to find the best resource settings and Java opts for Springboot applications running on Kubernetes? Should I give the application big resources and after some time, something like 7 days, lower it by maximal values on metrics?