Helm stable/nginx-ingress with autoscaling, HPA couldn't read CPU or memory metrics
I install the latest version (1.24.5) of stable/nginx-ingress with autoscaling: true on a GKE cluster.
After finish deploying, HPA said that unable to read all metrics.
Can you help me to debug and fix it? If there is any more detail/log do you need, please kindly ask and I will provide it!
Thank you
Update:
- I can run other HPA perfectly
kubectl get hpareturnsunknown/100m for CPU
unknown/100Mi for RAM
Similar Questions
3 Answers
I think that it could be that some of the pod’s containers do not have the relevant resource request set. Accordingly to the Kubernetes HPA documentation, when this happens the utilization for the pod can not be defined (and the autoscaler will not take any action for that metric).
You could do a quick check by running kubectl get hpa to list the current autoscalers. Or you could get a more detailed description with kubectl describe hpa. If you see some targets listed as <unknown> then it's most likely that the resource request was not set properly.
In case you have already set resources requests in for all your pods and you still see this message, maybe it would be helpful if you share more details about how you're defining requests and/or limits.
These articles from Kubernetes documentation could be useful:
Specify a memory request and memory limit
Specify a CPU request and a CPU limit
Regards!
I cannot comment yet. As marandall said in the GitHub issue Jackie linked above, this can be fixed by define resources limit and add --set controller.useComponentLabel=true --set defaultBackend.useComponentLabel=true to helm command.
It seems that since the HPA that is created by enabling the autoscaler on the ingress controller, it will apply the HPA on the label app=nginx-ingress. The nginx-ingress default-backend also has the same label, and thus also requires us to define the CPU/memory resource requests/limits as well.
You can find more info in this GitHub issue