HPA on deployment shows more memory utilization | Kubernetes

12/30/2019

I finally deployed hpa tied to one of the deployments, but hpa is not working as expected. I can see utilization is way beyond than what actually is, doesn't even match the sum of utilization across all pods. Not sure how this average utilization is been calculated, when with 2 pods of average memory 8gi shows 500%/85%, and don't even understand why the average utilization is showing more, 5*8gi. And I don't see the pods are scaling down.

NAME                                      REFERENCE                             TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/hpa   Deployment/task-deployment1   545%/85%, 1%/75%            2         5         5          36h

below is the hpa manifest file

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa
  namespace: namespace-CapAm
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: task-deployment1
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 75
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 85

any suggestions on why Mem utilization is wrong and where it might be coming from.

how can get the pod scale down worked.? tried increasing target threshold value which didn't helped

NAME                                      REFERENCE                             TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/hpa   Deployment/task-deployment1   545%/277%, 1%/75%            2         5         5          36h
-- lucifer758
docker
google-kubernetes-engine
horizontal-scaling
kubectl
kubernetes

1 Answer

1/2/2020

I have reproduced your issue base on CPU usage and system started to cool down gradually.

You may use "kubectl get hpa" time to time in order to observe the system behavior.

Please decrease load and check if you have same issue or not.

Please review documentation [1] which outlines how to configuring a horizontal pod Autoscaler.

[1] https://cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling

-- Pejvak
Source: StackOverflow