In AWS EKS, HPA (horizontal-pod-autoscaler) failed to get cpu utilization

9/14/2021

In order to apply Kubernetis for the first time as a backend developer of a startup, I looked for AWS' guide to EKS and found a good document and followed it.

The link to the guide is as follows. https://aws-eks-web-application.workshop.aws/en/10-intro.html

Here, I proceeded with the start with an AWS Account method on 2-1, and I'm omitting all the previous options.

ex. 5-2. (Option) Add Console Credential

In the initial attempt, we proceeded with the process with Option, because we continued to fail in the progress application stage and are trying new things.

All processes were easy until "10-1. Apply HPA stage".

However, when I checked the HPA Status through the kubectl get hpa command, CPU usage was marked as unknown.

The guide said that if you try a little later, it will come out properly, so I tried it an hour later, but it was the same.

So, when I checked the status through the kubectl describe hpa command, I found that there was an error due to the missing cpu request as below.

Name:                                                  demo-flask-backend-hpa
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Tue, 14 Sep 2021 09:03:53 +0000
Reference:                                             Deployment/demo-flask-backend
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 30%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: missing request for cpu
Events:
  Type     Reason                        Age   From                       Message
  ----     ------                        ----  ----                       -------
  Warning  FailedGetResourceMetric       5s    horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu
  Warning  FailedComputeMetricsReplicas  5s    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu

To solve this problem, we looked in many ways, but we haven't found a suitable solution because we still have little knowledge about Kubernetis.

The yaml setting files created so far are as follows.

All the instructions used in the Bashshell followed the guide, and there were no critical errors other than deprecated errors.

How could I solve this error?


flask-hpa.yaml

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: demo-flask-backend-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: demo-flask-backend
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 30

flask-deployment.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-flask-backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-flask-backend
  template:
    metadata:
      labels:
        app: demo-flask-backend
    spec:
      containers:
        - name: demo-flask-backend
          image: $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/demo-flask-backend:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
            limits:
              cpu: 500m

ingress.yaml

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "backend-ingress"
  namespace: default
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - path: /contents
            pathType: Prefix
            backend:
              service:
                name: "demo-flask-backend"
                port:
                  number: 8080
          - path: /services
            pathType: Prefix
            backend:
              service:
                name: "demo-nodejs-backend"
                port:
                  number: 8080
          - path: /
            pathType: Prefix
            backend:
              service:
                name: "demo-frontend"
                port:
                   number: 80
-- Jun
amazon-eks
amazon-web-services
kubernetes

2 Answers

9/14/2021

You need to install metrics-server to get cpu and memory metrics.

-- gohm&#39;c
Source: StackOverflow

9/14/2021

HPA works on the Metrics server data to scale POD or not.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

installation : https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html

In AWS you have to install it first, which in GKE it's come already installed by default.

https://aws.amazon.com/premiumsupport/knowledge-center/eks-metrics-server/

You can check if metrics server running or not using

kubectl top pods

if output comes will resources data and usage your metrics server is up & running there another issue with HPA.

-- Harsh Manvar
Source: StackOverflow