I am running different Java Containers in Kubernetes with OpenJDK 11.0.8 and Payara-Micro and Wildfly 20. Resources are defined as followed:

      resources:
        requests:
          memory: "512Mi"
        limits:
          memory: "1Gi"

OpenJDK is running with default memory settings which means a Heap Ratio of 25%.

So I assume that 1G is the upper limit of memory the container can consume. But after some days the upper limits are exceeded as the containers memory consumption increases slowly but steadily. In particular, it is the Non-Heap memory which is increasing and not the heap memory.

So I have two questions: why is the non-heap memory increasing and why is the container/jdk/gc ignoring the container memory limits?

Example Measurement

This is some example data about the pod and the memory consumption to illustrate the situation.

POD information (I skipped irrelevant data here):

$ kubectl describe pod app-5df66f48b8-4djbs -n my-namespace

Containers:
  app:
...
    State:          Running
      Started:      Sun, 27 Sep 2020 13:06:44 +0200
...
    Limits:
      memory:  1Gi
    Requests:
      memory:  512Mi
...
QoS Class:       Burstable

Check Memory usage with kubectl top:

-$ kubectl top  pod app-5df66f48b8-4djbs -n my-namespace
NAME                             CPU(cores)   MEMORY(bytes)   
app-587894cd8c-dpldq             72m          1218Mi

Checking memory limit inside the pod:

[root@app-5df66f48b8-4djbs jboss]# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
1073741824

VM Flags inside the pod/jvm:

[root@app-5df66f48b8-4djbs jboss]# jinfo  -flags <PID>
VM Flags:
-XX:CICompilerCount=2 -XX:InitialHeapSize=16777216 -XX:MaxHeapSize=268435456 -XX:MaxNewSize=89456640 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=5570560 -XX:NonNMethodCodeHeapSize=5825164 -XX:NonProfiledCodeHeapSize=122916538 -XX:OldSize=11206656 -XX:ProfiledCodeHeapSize=122916538 -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseSerialGC

Heap Info inside the POD/jvm:

[root@app-5df66f48b8-4djbs jboss]# jcmd <PID> GC.heap_info 
68:
 def new generation   total 78656K, used 59229K [0x00000000f0000000, 0x00000000f5550000, 0x00000000f5550000)
  eden space 69952K,  77% used [0x00000000f0000000, 0x00000000f34bfa10, 0x00000000f4450000)
  from space 8704K,  59% used [0x00000000f4cd0000, 0x00000000f51e7ba8, 0x00000000f5550000)
  to   space 8704K,   0% used [0x00000000f4450000, 0x00000000f4450000, 0x00000000f4cd0000)
 tenured generation   total 174784K, used 151511K [0x00000000f5550000, 0x0000000100000000, 0x0000000100000000)
   the space 174784K,  86% used [0x00000000f5550000, 0x00000000fe945e58, 0x00000000fe946000, 0x0000000100000000)
 Metaspace       used 122497K, capacity 134911K, committed 135784K, reserved 1165312K
  class space    used 15455K, capacity 19491K, committed 19712K, reserved 1048576K

VM Metaspace Info:

$ jcmd 68 VM.metaspace
68:

Total Usage - 1732 loaders, 24910 classes (1180 shared):
  Non-Class: 5060 chunks,    113.14 MB capacity,   104.91 MB ( 93%) used,     7.92 MB (  7%) free,     5.86 KB ( <1%) waste,   316.25 KB ( <1%) overhead, deallocated: 4874 blocks with 1.38 MB
      Class: 2773 chunks,     19.04 MB capacity,    15.11 MB ( 79%) used,     3.77 MB ( 20%) free,   256 bytes ( <1%) waste,   173.31 KB ( <1%) overhead, deallocated: 1040 blocks with 412.14 KB
       Both: 7833 chunks,    132.18 MB capacity,   120.01 MB ( 91%) used,    11.69 MB (  9%) free,     6.11 KB ( <1%) waste,   489.56 KB ( <1%) overhead, deallocated: 5914 blocks with 1.78 MB


Virtual space:
  Non-class space:      114.00 MB reserved,     113.60 MB (>99%) committed 
      Class space:        1.00 GB reserved,      19.25 MB (  2%) committed 
             Both:        1.11 GB reserved,     132.85 MB ( 12%) committed 



Chunk freelists:
   Non-Class:

 specialized chunks:   43, capacity 43.00 KB
       small chunks:   92, capacity 368.00 KB
      medium chunks: (none)
   humongous chunks: (none)
              Total:  135, capacity=411.00 KB
       Class:

 specialized chunks:   18, capacity 18.00 KB
       small chunks:   64, capacity 128.00 KB
      medium chunks: (none)
   humongous chunks: (none)
              Total:   82, capacity=146.00 KB

Waste (percentages refer to total committed size 132.85 MB):
              Committed unused:    128.00 KB ( <1%)
        Waste in chunks in use:      6.11 KB ( <1%)
         Free in chunks in use:     11.69 MB (  9%)
     Overhead in chunks in use:    489.56 KB ( <1%)
                In free chunks:    557.00 KB ( <1%)
Deallocated from chunks in use:      1.78 MB (  1%) (5914 blocks)
                       -total-:     14.62 MB ( 11%)


MaxMetaspaceSize: unlimited
CompressedClassSpaceSize: 1.00 GB

InitialBootClassLoaderMetaspaceSize: 4.00 MB

Answer

The reason for the high values was an incorrect output of the Metric data received from Project kube-prometheus. After uninstalling the kube-projemet and installing instead the metric-server all data was display correctly using kubctl top. It shows now the same values as docker stats. I do not know why kube-prometheus did compute wrong data. In fact it was providing the double values for all memory data. I will investigate in this issue.