I have a java application running in kubernetes pod with base image openjdk:11. And that pod restarts every 3-4 days because memory is running out. No OutOfMemoryException in logs, no nothing in logs about memory errors or about app termination. I know only than pod terminates with code 143 (Indicates failure as container received SIGTERM). I profiled with Java Flight Recorder in production and with VisualVM on local machine and have nothing facts about any anomaly. GC is working well, the heap is being cleared, the metaspace is not growing, threads are not increasing. But memory in pod is growing and ends with restart of pod. I monitor that pod with grafana and see next:
Pod started with next params: requests.memory=2000Mi and limits.memory=2600Mi
Java opts: -Xmx1300M -XX:CompressedClassSpaceSize=32M -XX:MaxMetaspaceSize=256M -XX:ReservedCodeCacheSize=128M -Xverify:none -Dspring.jmx.enabled=false -Djava.security.egd=file:/dev/./urandom -XX:+AlwaysActAsServerClassMachine -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions
Maybe someone will guide me on the right way. Thanks!
You have most of the usual problems covered ... maybe direct buffers??
It is known that in some scenarios the direct buffers might grow indefinitely effectively acting as a memory leak.