I'm looking for a Prometheus metric that would allow me to monitor the time pods spend in the terminating state before vanishing into the void.
I've tried playing around with kube_pod_container_status_terminated but it only seems to register pods once they finish the termination process, but don't help me understand how long does it take to terminate a pod.
I've also looked at kube_pod_status_phase which I found out about in this channel a while ago but it also seems to lack this insight.
I'm currently collecting metrics on my k8s workload using cAdvisor, kube-state-metrics and the prometheus node-exporter, but would happily considering additional collectors if they contain the desired data.
A non-prometheus solution would also be great.
Any ideas? Thanks!
Kubernetes itself, Heapster and metrics-server don't provide such metrics, but you can get metrics close to what you've mentioned by installing kube-state-metrics. It has several pod metrics that reflect pods state:
kube_pod_status_phase
kube_pod_container_status_terminated
kube_pod_container_status_terminated_reason
kube_pod_container_status_last_terminated_reason
You can find the full list of pods metrics, provided by kube-state-metrics in Documentation.
There is also Bitnami Helm chart that could simplify the installation of kube-state-metrics.