K
Q

kubernetes hpa cannot get cpu consumption

6/4/2016

I am trying to configure kubernetes horizontal pod autoscaling. The definition of my deployment is here.

I start heapster deployment and heapster service like below:

# kubectl run heapster --namespace=kube-system --image=gcr.io/google_containers/heapster:v1.0.2 --command -- /heapster --source=kubernetes:http://192.168.122.159:8080?inClusterConfig=false --sink=log
# kubectl expose deployment heapster --port=80 --target-port=8082 --name=heapster --namespace=kube-system

The hpa is create like this:

# kubectl autoscale deployment nginx-rc --max=5 --cpu-percent=25

The heapster seems to run fine. Here are the logs.

After all this, the hpa is not receiving any data on CPU utilization.

# kubectl describe hpa
Name:               nginx-rc
Namespace:          default
Labels:             <none>
Annotations:            <none>
CreationTimestamp:      Sat, 04 Jun 2016 07:23:03 -0400
Reference:          Deployment/nginx-rc/scale
Target CPU utilization:     25%
Current CPU utilization:    <unset>
Min replicas:           1
Max replicas:           5
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  9m        7s      22  {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 0/1 of pods
  9m        7s      22  {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/1 of pods

Controller manager logs have a recurring line:

Jun 04 07:21:33 master kube-controller-manager[8202]: W0604 07:21:33.079677    8202 horizontal.go:108] Failed to reconcile nginx-rc: failed to compute desired number of replicas based on CPU utilization for Deployment/default/nginx-rc: failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/1 of pods

Can someone point out the nature of the problem here?

-- Aditya Patawari
autoscaling
docker
kubernetes

Similar Questions

How do you ssh into a Google Compute Engine Disk?
What to use to orchestrate a few long running web services on few machines?
Generating new (not renewed) SSL Certificates with "letsencrypt certonly" in non-interactive mode
Debugging a NoDiskConflict in kubernetes

2 Answers

6/5/2016

I suspect this is because you're not running a heapster service. The metrics client for the HorizontalPodAutoscaler attempts to get the metrics from a service named heapster in the kube-system namespace. You can look at the default Heapster definition here.

If that's not the issue, look for /var/log/kube-controller-manager on the master node. In particular, look for entries with failed to get pods metrics.

-- Tim Allclair
Source: StackOverflow
6/8/2016

Most likely the problem is that you're not running DNS in your cluster.

HPA currently uses heapster service name to read metrics. You have properly created it, but HPA controller cannot resolve service name to IP address as there's not DNS in your cluster.

I have create issue https://github.com/kubernetes/kubernetes/issues/27044 to have more informative events that would help the user/admin understand the problem better.

-- Filip
Source: StackOverflow