`kubectl get hpa` showing targets as unknown and not autoscaling the pod when the load is increased?

9/12/2018

I create a deployment and one HPA for that deployment (single node minikube cluster). But when I run the kubectl get hpa, it shows the targets as unknown (screenshot below)

NAME       REFERENCE             TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
producer   Deployment/producer   <unknown>/1%   1         3         1          42m

Below is the deployment.yaml file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.1.0 (36652f6)
  creationTimestamp: null
  labels:
    io.kompose.service: producer
  name: producer
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: producer
    spec:
      containers:
      - image: <image name>
        name: producer
        ports:
        - containerPort: 8090
        env:
        - name: mongoUrl
          value: mongodb://mongo:27017
        - name: mongoHost
          value: mongo
        - name: mongoPort
          value: "27017"
        resources:
          requests:
            cpu: 10m
         resources: {}
       restartPolicy: Always
status: {}

I run kubectl autoscale deployment producer --cpu-percent=1 --min=1 --max=3 command for setting the autoscaler for the producer deployment. Below is the output for kubectl describe hpa :

Type     Reason                        Age                 From                       Message
  ----     ------                        ----                ----                       -------
  Warning  FailedComputeMetricsReplicas  43m (x13 over 49m)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu on container producer in pod default/producer-c7dd566f6-69gbq
  Warning  FailedGetResourceMetric       4m (x91 over 49m)   horizontal-pod-autoscaler  missing request for cpu on container producer in pod default/producer-c7dd566f6-69gbq

I have already enabled the heapster and metrics-server addons on minikube and waiting the hpa is running from the last 30 minutes.

Logs from kube-controller-manager in minikube:

{"log":"I0912 10:36:40.806224       1 event.go:218] Event(v1.ObjectReference{Kind:\"HorizontalPodAutoscaler\", Namespace:\"default\", Name:\"producer\", UID:\"135d0ebc-b671-11e8-a19f-080027646864\", APIVersion:\"autoscaling/v2beta1\", ResourceVersion:\"71101\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedGetResourceMetric' missing request for cpu on container producer in pod default/producer-c7dd566f6-w8zcd\n","stream":"stderr","time":"2018-09-12T10:36:40.80645916Z"}
{"log":"I0912 10:36:40.806511       1 event.go:218] Event(v1.ObjectReference{Kind:\"HorizontalPodAutoscaler\", Namespace:\"default\", Name:\"producer\", UID:\"135d0ebc-b671-11e8-a19f-080027646864\", APIVersion:\"autoscaling/v2beta1\", ResourceVersion:\"71101\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedComputeMetricsReplicas' failed to get cpu utilization: missing request for cpu on container producer in pod default/producer-c7dd566f6-w8zcd\n","stream":"stderr","time":"2018-09-12T10:36:40.806564094Z"}

Output from kubectl top nodes

NAME       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
minikube   336m         16%       1518Mi          19%    

I have tried setting the horizontal-pod-autoscaler-use-rest-clients=false for kube-controller-manager, still facing the same issue.

-- Lakshya Garg
kubernetes
minikube

1 Answer

9/17/2018

The problem was with my producer-deployment.yaml as I had overwritten the resources request to be null again by writing the line: resources: {} after

resources:
    requests:
        cpu: 10m

This made my resources request null and that is why heapster was showing missing request for cpu on container producer.

removing the line resources: {} form producer-deployment.yaml solved the issue.

-- Lakshya Garg
Source: StackOverflow