When your application in a pod reaches the limits of memory you set by resources.limits.memory or namespace limit, Kubernetes restarts the pod.

The Kubernetes part of limiting resources is described in the following articles:

• Kubernetes best practices: Resource requests and limits
• Resource Quotas
• Admission control plugin: ResourceQuota
• Assign Memory Resources to Containers and Pods

Memory consumed by Java application is not limited to the size of the Heap that you can set by specifying the options:

-Xmssize Specifies the initial heap size.
-Xmxsize Specifies the maximum heap size.

Java application needs some additional memory for metaspace, class space, stack size, and JVM itself needs even more memory to do its tasks like garbage collection, JIT optimization, Off-heap allocations, JNI code. It is hard to predict total memory usage of JVM with reasonable precision, so the best way is to measure it on the real deployment with usual load.

I would recommend you to set the Kubernetes pod limit to double Xmx size, check if you are not getting OOM anymore, and then gradually decrease it to the point when you start getting OOM. The final value should be in the middle between these points.
You can get more precise value from memory usage statistics in a monitoring system like Prometheus.

On the other hand, you can try to limit java memory usage by specifying the number of available options, like the following:

-Xms<heap size>[g|m|k] -Xmx<heap size>[g|m|k]
-XX:MaxMetaspaceSize=<metaspace size>[g|m|k]
-Xmn<young size>[g|m|k]
-XX:SurvivorRatio=<ratio>

More details on that can be found in these articles:

• Properly limiting the JVM’s memory usage (Xmx isn’t enough)
• Why does my Java process consume more memory than Xmx

The second way to limit JVM memory usage is to calculate heap size based on the amount of RAM(or MaxRAM). There is a good explanation of how it works in the article:

The default sizes are based on the amount of memory on a machine, which can be set with the -XX:MaxRAM=N flag. Normally, that value is calculated by the JVM by inspecting the amount of memory on the machine. However, the JVM limits MaxRAM to 1 GB for the client compiler, 4 GB for 32-bit server compilers, and 128 GB for 64-bit compilers. The maximum heap size is one-quarter of MaxRAM . This is why the default heap size can vary: if the physical memory on a machine is less than MaxRAM , the default heap size is one-quarter of that. But even if hundreds of gigabytes of RAM are available, the most the JVM will use by default is 32 GB: one-quarter of 128 GB. The default maximum heap calculation is actually this:

Default Xmx = MaxRAM / MaxRAMFraction

Hence, the default maximum heap can also be set by adjusting the value of the - XX:MaxRAMFraction=N flag, which defaults to 4. Finally, just to keep things interesting, the -XX:ErgoHeapSizeLimit=N flag can also be set to a maximum default value that the JVM should use. That value is 0 by default (meaning to ignore it); otherwise, that limit is used if it is smaller than MaxRAM / MaxRAMFraction .

The initial heap size choice is similar, though it has fewer complications. The initial heap size value is determined like this:

Default Xms = MaxRAM / InitialRAMFraction

As can be concluded from the default minimum heap sizes, the default value of the InitialRAMFraction flag is 64. The one caveat here occurs if that value is less than 5 MB —or, strictly speaking, less than the values specified by -XX:OldSize=N (which defaults to 4 MB) plus -XX:NewSize=N (which defaults to 1 MB). In that case, the sum of the old and new sizes is used as the initial heap size.

This article gives you a good point to start tuning your JVM for web-oriented application:

• Java VM Options You Should Always Use in Production

If you are able to run on Java 11 (or 10) instead of 8, the memory limit options have been much improved (plus the JVM is cgroups-aware). Just use -XX:MaxRAMPercentage (range 0.0, 100.0):

$ docker run -m 1GB openjdk:11 java -XshowSettings:vm -XX:MaxRAMPercentage=80 -version
VM settings:
    Max. Heap Size (Estimated): 792.69M
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Debian-2)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Debian-2, mixed mode, sharing)

That way, you can easily specify 80% of available container memory for the heap, which wasn't possible with the old options.

Thanks @VAS for your comments. Thanks for the kubernetes links.

After few tests I think that its not a good idea to specify XMX if you are using -XX:+UseCGroupMemoryLimitForHeap since XMX overrides it. I am still doing some more tests & profiling.

Since my requirement is running a JVM inside a docker container. I did few tests as mentioned in the posts by @Eugene. Considering every app running inside a JVM would need HEAP and some native memory, I think we need to specify -XX:+UnlockExperimentalVMOptions, XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=1 (considering only the JVM running inside the container, at the same time its risky) -XX:MaxRAM (I think we should specify this if MaxRAMFraction is 1 so that you leave some for native memory)

Few tests:

As per below docker configuration, the docker is allocated 1 GB considering you only have the JVM running inside the container. Considering docker's allocation to 1G and I also want to allocate some to the process/native memory, I think I should use MaxRam=700M so that I have 300 MB for native.

$ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XX:MaxRAM=700M -XshowSettings:vm -version VM settings: Max. Heap Size (Estimated): 622.50M Ergonomics Machine Class: server Using VM: OpenJDK 64-Bit Server VM

Now specifying XX:MaxRAMFraction=1 might be killing:

references: https://twitter.com/csanchez/status/940228501222936576?lang=en Is -XX:MaxRAMFraction=1 safe for production in a containered environment?

Following would be better, please note I have removed MaxRAM since MaxRAMFraction > 1 :

$ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -XshowSettings:vm -version VM settings: Max. Heap Size (Estimated): 455.50M Ergonomics Machine Class: server Using VM: OpenJDK 64-Bit Server VM

This gives rest of the 500M for native e.g. could be used for MetaSpace by specifying -XX:MaxMetaspaceSize:

$ docker run -m 1GB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -XX:MaxMetaspaceSize=200M -XshowSettings:vm -version VM settings: Max. Heap Size (Estimated): 455.50M Ergonomics Machine Class: server Using VM: OpenJDK 64-Bit Server VM

Logically and also as per the above references, it makes sense to specify -XX:MaxRAMFraction >1. This also depends on the application profiling done.

I am still doing some more tests, will update these results or post. Thanks