I'm using Kubernetes in a Google Container Engine cluster and I can see in the Stackdriver logs that one of the Pods is falling over and automatically restarting because of an OutOfMemory exception.
Is there any standard way of monitoring these events?
I'm using Heapster and it doesn't seem to provide any metrics about Pods shutting down and restarting.
According to our IT Sysadmin, most of the current solutions for real-time monitoring and alerting on pods failures are currently unstable or very lackluster.
We ended up developing a small script which uses the Slack.com webservice for mail notifications and such.
I am sorry if my answer is not a ready-to-use one click solution :-)
This is a real feedback from our current experience and searches.
I expect things will move fast in the near future on that topic!
Our code (simple solution for effective results):
There is a tool called kube-state-metrics (1), which provides metrics about k8s objects, including the restart count of containers (2). These metrics can be used by Prometheus (3), where you could create an alert if the restart count is higher than a specified number.
The Prometheus Operator of CoreOS (4) and their example configuration (5) might be useful if you decide to go this way, it was very helpful when we recently deployed it to our cluster. There is no predefined alert for the restart count in that example, but it should be easy to add one.
(1) https://github.com/kubernetes/kube-state-metrics
(2) https://github.com/kubernetes/kube-state-metrics/blob/master/Documentation/pod-metrics.md
(3) https://prometheus.io/
(4) https://coreos.com/operators/prometheus/docs/latest/user-guides/getting-started.html
(5) https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus