I would like to be able to programmatically query Kubernetes to find overcommitted nodes.
If I do kubectl describe nodes, I get human-readable output including information about resource usage that I'm after, e.g.
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 786m (40%) 5078m (263%)
memory 8237973504500m (74%) 13742432Ki (126%)
However, kubectl describe doesn't support JSON or YAML output, and kubectl get nodes -ojson doesn't include the allocated resource stats. Is there any other way to access this information?
You can write the output to a file maybe?
kubectl describe > output.txtIf you run any kubectl command with the --v=6 option the output will include the kubernetes API calls that make up the output.
In the case of kubectl describe nodes NODE you will see there is a api/v1/pods request that filters pods on the node and removes some "not running" statuses
I0828 13:44:29.310208 55233 round_trippers.go:438] GET https://kubernetes.docker.internal:6443/api/v1/pods?fieldSelector=spec.nodeName%3Ddocker-desktop%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded 200 OK in 4 millisecondsIf you complete this request with authentication information from your ~/.kube/config file you should be able to get the output. In this case, using jq to filter the output down to the resources component of the container spec with CA/Cert/Key auth (base64 decoded).
curl --cacert ~/.kube/docker-desktop.ca \
--cert ~/.kube/docker-desktop.cert \
--key ~/.kube/docker-desktop.key \
https://kubernetes.docker.internal:6443/api/v1/pods?fieldSelector=spec.nodeName%3Ddocker-desktop%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded \
| jq '.items[].spec.containers[].resources'{}
{}
{
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
}
{
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
}
{}
{
"requests": {
"cpu": "250m"
}
}
{
"requests": {
"cpu": "200m"
}
}
{}
{
"requests": {
"cpu": "100m"
}
}
{}
{}
Running these calls and filters will generally be easier with one of the kubernetes API clients if you are regularly going to this level.