Datadog: How to alert if a pod in running in kubernetes

12/21/2020

I'm using datadog to monitor the health of several pods deployed in a kubernetes cluster. I use a query like this to check the pods

avg(last_5m):avg:kubernetes.pods.running{environment:develop,kube_service:service} <= 0

If I stop the pod, there ins't any data for kubernetes.pods.running (so the value is not zero, I don't have any value) . I don't know if it's possible to check from datadog that no pods has kube_service running.

-- drizzt.dourden
datadog
kubernetes

1 Answer

12/21/2020

You can look in to other datadog kube metrics like kubernetes.replicas.available / total to alert if no of available - total < 0. Same can be done or for daemonset pods also there is a specific metric exposed. Datadog docs-kube metrics

-- Sunjay Jeffrish
Source: StackOverflow