Kubernetes HPA memory autoscale does not work

1/14/2019

As mentioned in doc, HPA apiVersion: autoscaling/v2beta1 support memory autoscale, but it does not work for me. Any help is highly appreciated.

HPA:

apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
      name: xxx
    spec:
      scaleTargetRef:
        apiVersion: extensions/v1beta1
        kind: Deployment
        name: xxx
        minReplicas: 1
        maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 70
    - type: Resource
      resource:
        name: memory
        targetAverageValue: 1050Mi`

Deployement.yml:
    `apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: xxx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        service: xxx
    spec:
      containers:
      - name: xxx
        image:  xxx
        imagePullPolicy: Always
        resources:
          requests:
            memory: "1024M"
            cpu: "500m"
          limits:
            memory: "2048M"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /swagger-ui.html
            port: 9002
          initialDelaySeconds: 5
          periodSeconds: 180
        readinessProbe:
          httpGet:
            path: /swagger-ui.html
            port: 9002
          initialDelaySeconds: 10
          periodSeconds: 5

$kubectl describe hpa

Metrics:                                               ( current / target )
  resource memory on pods:                             1343959859200m / 1050Mi
  resource cpu on pods  (as a percentage of request):  4% (28m) / 70%
Min replicas:                                          1
Max replicas:                                          10
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

$kubectl get hpa

NAME                    REFERENCE                          TARGETS                          MINPODS   MAXPODS   REPLICAS   AGE
xxx   Deployment/xxx   1551148373333m/1050Mi, 98%/70%   1         10        3          42m

$kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.5-gke.5", GitCommit:"9aba9c1237d9d2347bef28652b93b1cba3aca6d8", GitTreeState:"clean", BuildDate:"2018-12-11T02:36:50Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}

Gcloud information:

[compute]
    region: [us-east1]
    zone: [us-east1-b]

As displayed, K8S understand how much memory is over target value but does not autoscale pods even I have waited for a long time. I only auto-scale with CPU.

-- Tom Ho
google-kubernetes-engine

1 Answer

1/16/2019

You mention that the Autoscaler works with CPU only, when you deployed the HPA, did you only set the CPU metric?

After reading the HPA Walkthrough documentation in k8s, I noticed that when setting multiple metrics all of the them must be reached to scale, refer to this section:

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics

So in your case, the Autoscaler must meet both conditions the CPU and memory usage metric.

-- kornshell93
Source: StackOverflow