Kubernetes target cpu utilization policy

9/5/2018

I am unable to make the auto-scaling work with targetcpuutilization setting. My configuration is as follows:

apiVersion: autoscaling/v2beta1
  kind: HorizontalPodAutoscaler
  metadata:
  name: pod
  namespace: pod
spec:
  minReplicas: 1
  maxReplicas: 5
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: pod
metrics:
- type: Resource
resource:
  name: cpu
  targetAverageUtilization: 10
  message: 'the HPA was unable to compute the replica count: unable to get metrics
    for resource cpu: unable to fetch metrics from resource metrics API: the server
    could not find the requested resource (get pods.metrics.k8s.io)'
  reason: FailedGetResourceMetric

I have verified that the metrics server is running. When I check hpa I am getting the following:

NAME      REFERENCE           TARGETS         MINPODS   MAXPODS      REPLICAS   AGE 
pod-name    Deployment/pod-name   <unknown>/10%   1         5         1          15h

The event log for the namespace shows this:

LAST SEEN   FIRST SEEN   COUNT     NAME                      KIND                      SUBOBJECT   TYPE      REASON                         SOURCE                      MESSAGE
46m         15h          1721      pod-name.155162c884d417be   HorizontalPodAutoscaler               Warning   FailedComputeMetricsReplicas   horizontal-pod-autoscaler   failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
1m          15h          1811      pod-name.155162c884a5caa2   HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: no metrics returned from resource metrics API

Looks like the pods are not able to get to the metrics server.

Logs on the metrics server:

unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:host.com: unable to fetch metrics from Kubelet host.com (host.com): Get https://host.com:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source

  kubelet_summary:host.ibaset.com: unable to fetch metrics from Kubelet host.ibaset.com (host.ibaset.com): Get https://host.com:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:host.com: unable to fetch metrics from Kubelet host.com (host.com): Get https://host.com:10250/stats/summary/: x509: certificate signed by unknown authority]
-- sharman
autoscaling
kubectl
kubernetes
metrics

1 Answer

9/5/2018

It looks like the metrics server can't receive metrics from the Kubelet endpoint because of an invalid certificate.

Try adding following argument to the metrics-server: --kubelet-insecure-tls

-- bn4t
Source: StackOverflow