my containers run in kubernetes and I saw lots of OOM from kubernetes Node log. All the logs only have process id information. it's hard to dig what containers OOM. I don't know how to find container id by the host process Id.
Is there any way i can get host process id within container so that i can get a mapping.
this is node log
2020-04-28 09:27:15.530 HKT
I0428 01:27:15.530763 1627 log_monitor.go:115] New status generated: &{Source:kernel-monitor Events:[{Severity:warn Timestamp:2020-04-28 01:27:08.060896434 +0000 UTC m=+89600.088785273 Reason:OOMKilling Message:Memory cgroup out of memory: Kill process 2493556 (node) score 1432 or sacrifice child
2020-04-28 09:29:15.000 HKT
Memory cgroup out of memory: Kill process 2493562 (node) score 1529 or sacrifice child Killed process 2493562 (node) total-vm:14009952kB, anon-rss:3146688kB, file-rss:28720kB, shmem-rss:0kB
2020-04-28 09:29:15.000 HKT
Memory cgroup out of memory: Kill process 2496433 (node) score 1275 or sacrifice child Killed process 2496433 (node) total-vm:7183684kB, anon-rss:1833580kB, file-rss:28804kB, shmem-rss:0kB
2020-04-28 09:29:15.309 HKT
I0428 01:29:15.309829 1627 log_monitor.go:115] New status generated: &{Source:kernel-monitor Events:[{Severity:warn Timestamp:2020-04-28 01:29:07.829961434 +0000 UTC m=+89719.857850273 Reason:OOMKilling Message:Memory cgroup out of memory: Kill process 2493562 (node) score 1529 or sacrifice child
2020-04-28 09:29:15.330 HKT
I0428 01:29:15.329925 1627 log_monitor.go:115] New status generated: &{Source:kernel-monitor Events:[{Severity:warn Timestamp:2020-04-28 01:29:07.849907434 +0000 UTC m=+89719.877796273 Reason:OOMKilling Message:Memory cgroup out of memory: Kill process 2496433 (node) score 1275 or sacrifice child
2020-04-28 09:48:29.000 HKT
Memory cgroup out of memory: Kill process 3086395 (monitor) score 237 or sacrifice child Killed process 3086395 (monitor) total-vm:130128kB, anon-rss:9204kB, file-rss:15488kB, shmem-rss:0kBAlright, this is a design error that looks to me. I would suggest using 1 container - 1 process and let the process become the main process (PID 1). This design will tell you which pod is having issues since Kubernetes will restart the pod as soon as the process hits the memory limit. Anyway, using Kubernetes you should see which pod has been restarted using kubectl get pods and see either pods in error start or restarting counter increasign. If you want to use only node logs, which I discorage you to use it, you might find that this solution could help you CoreOS - get docker container name by PID?