Horizontal auto scaler doesn't report CPU usage

9/13/2017

I am trying to test the Kubernetes HPA like in example here

kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
kubectl autoscale deployment php-apache --cpu-percent=20 --min=1 --max=10
kubectl get hpa

output is

NAME         REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown> / 20%   1         3         0          1h

Why isn't controller-manager able to get current usage?


Environment details :

  • K8s version 1.7.3 with Heapster running with a service on port 80, in kube-system.
  • InfluxDB is set as sink and I can see the usage/limit etc. correctly in Grafana as well as on Heapster API
  • I can see the cpu requests (200m) on the deployed pod's yaml.
  • No errors in controller manager logs. All I see is etcd watch and pod creation messages.
  • There is no "horizontal-pod-autoscaler-sync-period" specified in controller manifest, default should be 30 seconds.
  • "hpa describe" doesn't list any events ("<none>").
  • Creating/deleting HPA doesn't create any log in controller-manager
  • When I didn't have Heapster installed, controller-manager logs would complain. I do not see this message with Heapster installed : "

failed to compute desired number of replicas based on listed metrics for Deployment/xx/php-apache: failed to get cpu utilization: unable to get metrics for resource cpu: failed to get pod resource metrics: the server cannot complete the requested operation at this time, try again later (get services http:heapster:)


Edit 1 [14-Sep] :

  • I have uploaded the Heapster logs here. No errors there.
  • I checked the Heapster API directly for metrics, and I get response.

    curl http://10.107.55.59/api/v1/model/namespaces/cp/pods/php-apache-593471247-c65fh/metrics/cpu/usage

{ "metrics": [ { "timestamp": "2017-09-14T13:46:00Z", "value": 19008838 }, { "timestamp": "2017-09-14T13:47:00Z", "value": 19008838 }, { "timestamp": "2017-09-14T13:48:00Z", "value": 19008838 } ], "latestTimestamp": "2017-09-14T13:48:00Z" }

Grafana Screenshot for the pod. No limit was set, but that doesn't make any difference either.

-- Paddy
heapster
kubernetes

1 Answer

9/14/2017

Validate if heapster is fetching data and giving metric, try to run below on kube-api server, replace namespace-name and pod-name in it. it will show metric, if all well, also paste heapster log here

curl http://localhost:8080/api/v1/model/namespaces/{namespace-name}/pods/{pod-name}/metrics/

-- Pawan Kumar
Source: StackOverflow