I am trying to set up alerts/monitors for my Kubernetes PODs where my application is deployed.

I am using Kubernetes=1.15.

I am trying to set up alert for following two scenarios: 1. When any POD goes down due to some reason, alert should get triggered. 2. When my application goes down on any of the POD, alert should get triggered.

For (1) I am setting up an alert if Liveness probe failed >= 1, if all you care about is one or more pods going down

For (2), I am setting up an alert on Readiness probe_type and result as failed >= 1

For (1), I am using kubelet:prober_probe_total{probe_type=Readiness,result=failed,..}

For (2), I am using kubelet:prober_probe_total{probe_type=Liveness,result=failed,..}

Reference:

1. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/prober/prober_manager.go

2. https://godoc.org/k8s.io/kubernetes/pkg/kubelet/prober

I have been thinking that an alert should get triggered when I get any non zero value for them. But even when everything is fine, I am getting non-zero data.

So I am clueless here.