I am unable to make the auto-scaling work with targetcpuutilization setting. My configuration is as follows:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: pod
namespace: pod
spec:
minReplicas: 1
maxReplicas: 5
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: pod
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 10
message: 'the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)' reason: FailedGetResourceMetric
I have verified that the metrics server is running. When I check hpa I am getting the following:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE pod-name Deployment/pod-name <unknown>/10% 1 5 1 15h
The event log for the namespace shows this:
LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
46m 15h 1721 pod-name.155162c884d417be HorizontalPodAutoscaler Warning FailedComputeMetricsReplicas horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
1m 15h 1811 pod-name.155162c884a5caa2 HorizontalPodAutoscaler Warning FailedGetResourceMetric horizontal-pod-autoscaler unable to get metrics for resource cpu: no metrics returned from resource metrics API
Looks like the pods are not able to get to the metrics server.
Logs on the metrics server:
unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:host.com: unable to fetch metrics from Kubelet host.com (host.com): Get https://host.com:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source
kubelet_summary:host.ibaset.com: unable to fetch metrics from Kubelet host.ibaset.com (host.ibaset.com): Get https://host.com:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:host.com: unable to fetch metrics from Kubelet host.com (host.com): Get https://host.com:10250/stats/summary/: x509: certificate signed by unknown authority]
It looks like the metrics server can't receive metrics from the Kubelet endpoint because of an invalid certificate.
Try adding following argument to the metrics-server: --kubelet-insecure-tls