K
Q

kubernetes HPA deployment cannot find target resource

12/22/2019

I'm trying to deploy my own HPA without a success. While Trying to deploy the official PHP of kubernetes, it works as planned. When I'm trying to deploy my own HPA deployment file (on Nodejs), it doesn't work.

The process:

$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
service "php-apache" created
deployment "php-apache" created

My HPA deployment yaml:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.spec.replicaCountMin }}
  maxReplicas: {{ .Values.spec.replicaCountMax }}
  targetCPUUtilizationPercentage: 50
  selector:
    matchLabels:
      app: {{ .Values.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
        release: {{ .Release.Name }}
        heritage: {{ .Release.Service }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

The result:

PHP works as planned

NAME        REFERENCE              TARGETS  MINPODS   MAXPODS   REPLICAS   AGE
php-apache  Deployment/php-apache  0%/50%    1        3         1          3d7h

My release doesn't

NAME              REFERENCE                    TARGETS         MINPODS  MAXPODS   REPLICAS   AGE
gw-autoscale-t6   Deployment/gw-autoscale-t6   <unknown>/80%   1        3         0          11m

My release HPA Json

Name:                                                  gw-autoscale-t6
Namespace:                                             dev
Labels:                                                app=gw-autoscale-t6
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 22 Dec 2019 22:39:44 +0200
Reference:                                             Deployment/gw-autoscale-t6
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 80%
Min replicas:                                          1
Max replicas:                                          3
Deployment pods:                                       0 current / 0 desired
Conditions:
  Type         Status  Reason          Message
  ----         ------  ------          -------
  AbleToScale  False   FailedGetScale  the HPA controller was unable to get the target's current scale: deployments/scale.apps "gw-autoscale-t6" not found
Events:
  Type     Reason          Age                 From                       Message
  ----     ------          ----                ----                       -------
  Warning  FailedGetScale  27s (x81 over 20m)  horizontal-pod-autoscaler  deployments/scale.apps "gw-autoscale-t6" not found

  • I have also tried a lot of other types of deployment files.

  • My Metrics-Service is installed.

  • I'm installing my deployment with Helm by Nodejs project.

What can I do to solve this one?

Response for questions:

Deployment result

{
   "apiVersion": "autoscaling/v1",
   "kind": "HorizontalPodAutoscaler",
   "metadata": {
       "annotations": {
           "autoscaling.alpha.kubernetes.io/conditions": "[{\"type\":\"AbleToScale\",\"status\":\"False\",\"lastTransitionTime\":\"2019-12-22T20:39:59Z\",\"reason\":\"FailedGetScale\",\"message\":\"the HPA controller was unable to get the target's current scale: deployments/scale.apps \\\"gw-autoscale-t6\\\" not found\"}]"
       },
       "creationTimestamp": "2019-12-22T20:39:44Z",
       "labels": {
           "app": "gw-autoscale-t6"
       },
       "name": "gw-autoscale-t6",
       "namespace": "dev",
       "resourceVersion": "17299134",
       "selfLink": "/apis/autoscaling/v1/namespaces/dev/horizontalpodautoscalers/gw-autoscale-t6",
       "uid": "2f7e014c-24fb-11ea-a4d8-a28620329da6"
   },
   "spec": {
       "maxReplicas": 3,
       "minReplicas": 1,
       "scaleTargetRef": {
           "apiVersion": "apps/v1",
           "kind": "Deployment",
           "name": "gw-autoscale-t6"
       },
       "targetCPUUtilizationPercentage": 80
   },
   "status": {
       "currentReplicas": 0,
       "desiredReplicas": 0
   }
}
-- bloodix
kubectl
kubernetes
kubernetes-helm
node.js

Similar Questions

What happens to traffic to a temporary unavailable pod in a StatefulSet?
Is a Kubernetes service cluster specific or can a service span across clusters?
Helm install locally but upgrade in pipeline?
Unable to build docker image from docker file, for dotnet core web application
How does Kubernetes implement Linux capabilities?
IBM Websphere on kubernetes pod
How to add harbor registry in containerd

2 Answers

2/16/2020

The way I solved it is by adding the cpu resources to the deployment's file and keeping only the necessary HPA deployment yaml fields.

HPA deployment file

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.spec.replicaCountMin }} 
  maxReplicas: {{ .Values.spec.replicaCountMax }} 
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.spec.cpuUtil }}

Deployment file - resource addition

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: {{ .Values.name }}
  labels:
    app: {{ .Values.name }}
  namespace: {{ .Values.namespace }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
        release: {{ .Release.Name }}
        heritage: {{ .Release.Service }}
    spec:
      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        resources:
          requests:
            cpu: {{ .Values.request.cpu }}
          limits:
            cpu: {{ .Values.limits.cpu }}

You must add this code:

resources:
requests:
cpu: {{ .Values.request.cpu }}
limits:
cpu: {{ .Values.limits.cpu }}

Hope it helps :)

-- bloodix
Source: StackOverflow
12/31/2019

It looks like something is wrong with the deployment.

I have tried to reproduce the situation and it looks like you are creating autoscaler from file with kubectl create -f hpa.yaml that looks for the Deployment/gw-autoscale-t6 that is not present in cluster.

That is why you are getting the following error:

the HPA controller was unable to get the target''s current scale: deployments/scale.apps\ "gw-autoscale-t6" not found"

for my tests I've created a hpa.yaml, applied it and looked for the situation:

$ date && kubectl get hpa && kubectl get deployments php-apache 

Tue 31 Dec 2019 10:59:37 AM CET

NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/50%   1         10        0          10m

Error from server (NotFound): deployments.extensions "php-apache" not found

it gave me <unknown> as a TARGETS value. And Zero as REPLICAS

As you can see, the Deployment itself (the one we are trying to scale with hpa.yaml) doesn't exist yet.

Upon creation of a deployment:

$ kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80

service/php-apache created
deployment.apps/php-apache created

I have waited for a minute to let HPA to warm-up and calculate everything:

$ kubectl get hpa -o yaml

apiVersion: v1
items:
- apiVersion: autoscaling/v1
  kind: HorizontalPodAutoscaler
  metadata:
    annotations:
      autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2019-12-31T09:59:47Z","reason":"ScaleDownStabilized","message":"recent
        recommendations were higher than current one, applying the highest recent
        recommendation"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2019-12-31T10:00:33Z","reason":"ValidMetricFound","message":"the
        HPA was able to successfully calculate a replica count from cpu resource utilization
        (percentage of request)"},{"type":"ScalingLimited","status":"False","lastTransitionTime":"2019-12-31T10:00:33Z","reason":"DesiredWithinRange","message":"the
        desired count is within the acceptable range"}]'
      autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":0,"currentAverageValue":"1m"}}]'
    creationTimestamp: "2019-12-31T09:49:16Z"
    name: php-apache
...
 spec:
    maxReplicas: 10
    minReplicas: 1
    scaleTargetRef:
      apiVersion: extensions/v1beta1
      kind: Deployment
      name: php-apache
    targetCPUUtilizationPercentage: 50
  status:
    currentCPUUtilizationPercentage: 0
    currentReplicas: 1
    desiredReplicas: 1

And checked again:

$ date && kubectl get hpa && kubectl get deployments php-apache 

Tue 31 Dec 2019 11:01:16 AM CET
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          12m

NAME         READY   UP-TO-DATE   AVAILABLE   AGE
php-apache   1/1     1            1           97s

So now, I can see non-zero value for REPLICA and TARGETS shows correct value.

Hope that helps :) P.S. Let me know if you have solved the issue.

-- Nick
Source: StackOverflow