Active memory consumption by kubernetes pod with java app (not OutOfMemoryException)

8/12/2021

I have a java application running in kubernetes pod with base image openjdk:11. And that pod restarts every 3-4 days because memory is running out. No OutOfMemoryException in logs, no nothing in logs about memory errors or about app termination. I know only than pod terminates with code 143 (Indicates failure as container received SIGTERM). I profiled with Java Flight Recorder in production and with VisualVM on local machine and have nothing facts about any anomaly. GC is working well, the heap is being cleared, the metaspace is not growing, threads are not increasing. But memory in pod is growing and ends with restart of pod. I monitor that pod with grafana and see next:

enter image description here

Pod started with next params: requests.memory=2000Mi and limits.memory=2600Mi

Java opts: -Xmx1300M -XX:CompressedClassSpaceSize=32M -XX:MaxMetaspaceSize=256M -XX:ReservedCodeCacheSize=128M -Xverify:none -Dspring.jmx.enabled=false -Djava.security.egd=file:/dev/./urandom -XX:+AlwaysActAsServerClassMachine -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions

Maybe someone will guide me on the right way. Thanks!

-- user2625402
java
kubernetes
out-of-memory

1 Answer

12/1/2021

You have most of the usual problems covered ... maybe direct buffers??

It is known that in some scenarios the direct buffers might grow indefinitely effectively acting as a memory leak.

https://www.evanjones.ca/java-bytebuffer-leak.html

-- Rafael
Source: StackOverflow