I am trying to test the horizontal pod autoscale feature for my Kubernetes cluster deployed in AWS(using EKS).
I have set Resource as 'CPU' and target type as 'Utilization' and set 'averageUtilization' to 15 in the YAML file.
So technically when the CPU utilization percentage meets the 15% set percentage, Kubernetes cluster should auto-scale horizontally(add pods automatically). This is an important part of the YAML file:
maxReplicas: 11
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 15I am pumping a lot of HTTP traffic into the Kubernetes cluster, but all pods stabilize at 35% - 36% CPU with 5 or 6 pods and they do not create additional pods to bring down average CPU utilization to 15% (which I set in YAML file as 'averageUtilization').
No matter how much I debug and run tests many times, it's the same case. What am I doing wrong here, am I missing something else?
Did you check that metrics-server is enabled? It is required in order to enable hpa: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-metrics-apis.
Here's metrics-server doc: https://github.com/kubernetes-sigs/metrics-server. In order to test it quickly and check it solves your issue, you can disable TLS: --kubelet-insecure-tls. In production, you'll need to enable TLS again.
To check that metrics-server is enabled, you can run the kubectl top command. It requires a few seconds to start, because it needs to collect some metrics first.