How to track Kubernetes OOMKilled termination messages?

2/22/2018

When a POD is evicted is suposed to find messages about the eviction at /dev/termination-log (or other path defined in "terminationMessagePath" property) inside the container filesystem.

I'm trying to track OOMKilled events, but after OOMKilled occured and the POD was restarted, the /dev/termination-log is empty.

I tried to set "terminationMessagePolicy" property to "FallbackToLogsOnError" hoping to find out messages in container log and I did not find anything about the OOMKilled there.

By executing kubectl describe pod podname I can find out that the termination reason was OOMKilled, but I would like to be able to follow this events in order to integrate it with fluentd and logstash.

Our Kubernetes cluster version is v1.9.0 and it's running on-premise. The installation was done using kubeadm on top of CentOS 7.

-- Matheus Neder
kubernetes

2 Answers

11/9/2018

The way I'm using - is looking to dmesg on nodes and track events from there

-- Кирилл Семаев
Source: StackOverflow

2/22/2018

You can ssh to the machine the pod was running and exec journalctl -u kubelet

-- Cindy
Source: StackOverflow