I would like to understand when pods are in a pending state because required resources aren't available to them to be scheduled..Is there a way to track time that pods are spending in "Pending" or "Scheduled" states?
To display all pods - also in permanent "pending" state you can use:
kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.status.startTime}{"\n"}{end}'
kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:metadata.namespace,POD:metadata.name,STATE:status.containerStatuses[*].state.waiting.reason,PHASE:status.phaseresources:
lastTransitionTime under the status.conditions field in the pod manifest shows the timestamp of each intermediate state of the pod before it reaches Running state.
Timestamps below of the pod's transition Initialized -> Ready -> ContainersReady -> PodScheduled:
$ kubectl get pod coredns-fb8b8dccf-2lhl4 -n=kube-system -o json | jq '.status.conditions'
[
{
"lastProbeTime": null,
"lastTransitionTime": "2019-07-22T19:58:17Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2019-07-22T19:58:56Z",
"status": "True",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2019-07-22T19:58:56Z",
"status": "True",
"type": "ContainersReady"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2019-07-22T19:58:17Z",
"status": "True",
"type": "PodScheduled"
}
]
After a pod is initialized, it is usually in Pending state until it is scheduled (PodScheduled state above) and reaches Running state.
There are no other answers yet...so here we go:
kubectl -n <namespace> get events --sort-by=.metadata.creationTimestamp...will list events sorted by timestamp (replace <namespace> as appropriate).
For a concrete pod, this will output comma-delimited list of status changes and their timestamps:
kubectl -n <namespace> get po/<pod-name> -o jsonpath="{range .status.conditions[*]}{.type}{','}{.lastTransitionTime}{'\n'}{end}"From those, considering Pod Lifecycle, and a little scripting, it should be possible to derive the info you are looking for.