I finally deployed hpa tied to one of the deployments, but hpa is not working as expected. I can see utilization is way beyond than what actually is, doesn't even match the sum of utilization across all pods. Not sure how this average utilization is been calculated, when with 2 pods of average memory 8gi shows 500%/85%, and don't even understand why the average utilization is showing more, 5*8gi. And I don't see the pods are scaling down.
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/hpa Deployment/task-deployment1 545%/85%, 1%/75% 2 5 5 36h
below is the hpa manifest file
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: hpa
namespace: namespace-CapAm
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: task-deployment1
minReplicas: 2
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 75
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 85
any suggestions on why Mem utilization is wrong and where it might be coming from.
how can get the pod scale down worked.? tried increasing target threshold value which didn't helped
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/hpa Deployment/task-deployment1 545%/277%, 1%/75% 2 5 5 36h
I have reproduced your issue base on CPU usage and system started to cool down gradually.
You may use "kubectl get hpa" time to time in order to observe the system behavior.
Please decrease load and check if you have same issue or not.
Please review documentation [1] which outlines how to configuring a horizontal pod Autoscaler.
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/horizontal-pod-autoscaling