K
Q

metrics not available yet metrics-server kubernetes on aws

6/12/2019

I am trying to enable Metrics Server on AWS, and I followed these steps

  1. Clone or download the Metrics Server project.

  2. Open the deploy/1.8+/metrics-server-deployment.yaml file in an editor.

  3. Add the following command values into the containers property (it should be at the same level as the image property).

command: - /metrics-server - --kubelet-insecure-tls

  1. Run kubectl create -f deploy/1.8+ as shown on the Metrics Server repo to create the deployment, services, etc.

Till this point everything is working fine, my metrics-server-pod is running fine, but when I do kubectl top nodes I am getting the following error

error: metrics not available yet

when i did kubectl logs [metrics-server-pod-name] -n kube-system i am getting this

E0611 14:36:57.527048 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-20-xx-xxx.ec2.internal": no metrics known for node

E0611 14:36:57.527069 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-20-xx-xx.ec2.internal": no metrics known for node

E0611 14:36:57.527075 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-xx-114-xxx.ec2.internal": no metrics known for node

E0611 14:36:57.527079 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-xx-xx-xxx.ec2.internal": no metrics known for node

E0611 14:36:57.527084 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-xx-91-xx.ec2.internal": no metrics known for node

E0611 14:36:57.527088 1 reststorage.go:129] unable to fetch node metrics for node "ip-172-20-xx-xxx.ec2.internal": no metrics known for node

E0611 14:37:26.006830 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-172-xx-36-103.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-36-103.ec2.internal (172.20.36.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xx-65-xx.ec2.internal: unable to fetch metrics from Kubelet ip-172-20-65-xxx.ec2.internal (172.xx.65.xx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xx-114-xxx.ec2.internal: unable to fetch metrics from Kubelet ip-172-20-114-223.ec2.internal (172.xx.114.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xx-63-xxx.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-63-xxx.ec2.internal (172.xx.63.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xxx-91-xx.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-91-xxx.ec2.internal (172.xx.91.xxx): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:ip-172-xxx-96-xxx.ec2.internal: unable to fetch metrics from Kubelet ip-172-xx-96-xxx.ec2.internal (172.xx.96.xxx): request failed - "401 Unauthorized", response: "Unauthorized"]
-- Deepchand Swami
kubernetes

Similar Questions

NFS server on Kubernetes minikube reports `exportfs: / does not support NFS export`
How to override dnsconfig in kubernetes deployment kind
Is there a way in Kubernetes to check when hpa happened?
One node for a GKE cluster cannot pull image from dockerhub
Istio. How to add Virtual Host to Envoy?

1 Answer

6/14/2019

Option I.

Add - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname

to your metrics-server-deployment.yaml, e.g.

command:
  - /metrics-server
  - --kubelet-insecure-tls
  - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname

If still doesn't work, try with

Option II.

Because of metrics-server resolving the hostname from coredns, add node IPs to coredns configmap

kubectl edit configmap coredns -n kube-system

and add

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        hosts {
           192.168.199.100 master.qls.com
           192.168.199.220 worker.qls.com
           fallthrough
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2019-05-17T12:32:08Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "180"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: c93e5274-789f-11e9-a0ea-42010a9c0003
-- A_Suh
Source: StackOverflow