This is the first question for me, I'm sorry if I'm not giving all the necessary information.

This is the situation,
first time on AWS, first time on K8s...
we have a microservice infrastructure,
we have memory issue with pods/container with jdk8 so we move to jdk11-openj9 memory management if far way well than before, the issue now is a strange one, a microservice, that is working fine with the assigned memory, get OOMKILL during the not working hours, at random hours, today ar 5.03 AM this POD is configured to have a QOS = GUARANTEED and this is the only one so EVICTION is not what is happening.
We monitored (with grafana) also the other PODS in that NODE and at the killing time, no one has a memory spike

bitly (the one killed): [scrennshot from grafana]

config: [scrennshot from grafana]

oauth2: [scrennshot from grafana]

redis: [scrennshot from grafana]

customactivity: [scrennshot from grafana]

chat: [scrennshot from grafana]

someone have a suggestion where to look?

OOMKILLED sure not for memory excessive request in my opinion.

update 15/05/2020

killed again this morning : State: Running Started: Fri, 15 May 2020 01:11:27 +0000 Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Wed, 13 May 2020 07:32:46 +0000 Finished: Fri, 15 May 2020 01:11:27 +0000 Ready: True Restart Count: 1 Limits: memory: 250Mi Requests: cpu: 100m memory: 250Mi

no reason why OOMKILLED, no memory peak or system overload