I'm trying to calculate the average time a pod stays in a pending state in grafana with prometheus. I can generate a graph to get the number of pods in a pending state over time, with this query
sum(kube_pod_status_phase{phase="Pending"})However, I would really like to get an the value of the average time that the pods are staying in this state in the last X hours. How can I do that?
The metric kube_pod_status_phase{phase="Pending"} will only give you binary values i.e., 0/1. 1 if the pod is in pending state, otherwise. Also, the data is being updated updated every 30s. So to find the total time it was in pending in last X hours, you can do something like.
sum_over_time(kube_pod_status_phase{phase="Running"}[Xh]) * X * 30For better visualization, you can use table in grafana.
PromQL provides aggregation functions over time.
You specify the range of time in the expression ; for X previous hours, it will be:
avg_over_time(kube_pod_status_phase{phase="Pending"}[Xh])