On a Kubernetes cluster with CoreOS Prometheus Operator scraping all standard cluster metrics, what Prometheus metric would show me the currentCPUUtilizationPercentage
value for a simple HPA (horizontal pod autoscaler)?
If I setup a simple hpa like:
$ kubectl autoscale deployment php-apache --cpu-percent=30 --min=3 --max=10
And then if I do kubectl get hpa php-apache -o yaml
I see something like:
spec:
maxReplicas: 10
minReplicas: 3
targetCPUUtilizationPercentage: 30
...
status:
currentCPUUtilizationPercentage: 28
currentReplicas: 9
desiredReplicas: 9
...
I want to see that currentCPUUtilizationPercentage
in Prometheus. I've done a bunch of Prometheus queries to look for this.
I've searched all metrics tagged {hpa="php-apache"}
and I see many hpa metrics, but not the metric I'm looking or. I can see kube_hpa_spec_target_metric
set to 30 and I can see current status metrics like kube_hpa_status_condition
but not the current cpu metric value that I want to see.
I've searched all metrics tagged {metric_name="cpu"}
and only see kube_hpa_spec_target_metric
I've searched all container and pod related metrics tagged {container="my-container-name"}
and {pod=~"my-pod-prefix.*"}
and I see several cpu related metrics like container_cpu_usage_seconds_total
abnd container_spec_cpu_quota
but nothing similar to or nothing that seems to be able to be used in calculating the currentCPUUtilizationPercentage
value that I'm looking for?
FYI, this is on Kubernetes 1.17.x and using a recent version of CoreOS Prometheus Operator.
If I remember correctly currentCPUUtilizationPercentage is a k8 internal metrics for HPA and not exposed directly as a metrics you can scrape with Prometheus. see https://github.com/kubernetes/community/blob/master/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#autoscaling-algorithm
you probably could scrape component of currentCPUUtilizationPercentage metrics and create custom metrics to see it in Prometheus.