K
Q

Kubernetes Metrics unable to fetch pod/node metrics

9/7/2018

I've installed metrics-server on kubernetes v1.11.2.

I'm running a bare-metal cluster using 3 nodes and 1 master

In the metrics-server log I have the following errors:

E0907 14:29:51.774592       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:vps01: unable to 
fetch metrics from Kubelet vps01 (vps01): Get https://vps01:10250/stats/summary/: dial tcp: lookup vps01 on 10.96.0.10:53: no such host, unable to fully scr
ape metrics from source kubelet_summary:vps04: unable to fetch metrics from Kubelet vps04 (vps04): Get https://vps04:10250/stats/summary/: dial tcp: lookup 
vps04 on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:vps03: unable to fetch metrics from Kubelet vps03 (vps03): 
Get https://vps03:10250/stats/summary/: dial tcp: lookup vps03 on 10.96.0.10:53: no such host, unable to fully scrape metrics from source kubelet_summary:vp
s02: unable to fetch metrics from Kubelet vps02 (vps02): Get https://vps02:10250/stats/summary/: dial tcp: lookup vps02 on 10.96.0.10:53: no such host]     
E0907 14:30:01.694794       1 reststorage.go:98] unable to fetch pod metrics for pod boxweb/boxweb-deployment-7756c49688-fz625: no metrics known for pod "bo
xweb/boxweb-deployment-7756c49688-fz625"                                                                                                                    
E0907 14:30:10.517886       1 reststorage.go:112] unable to fetch node metrics for node "vps01": no metrics known for node "vps01"

I also can't get any metrics using kubectl top node vps01

Same for autoscale it is not working

  unable to get metrics for resource cpu: unable to fetch metrics from
 resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
-- Ben D
kubernetes

Similar Questions

Can we configure kube-router network policy from Azure Kubernetes Service using ARM Template
Kubernetes + Jenkins: how to assign all jenkins slave to one specific node
Google stackdriver debug not working in Kubernetes
Scaling with GPU usage
Error 403 Forbidden to metrics server connect with nodes
k3s redirect http to https

3 Answers

6/14/2019

I got the same issue and I resolved by adding hostname in /etc/hosts on every node.

For collecting metric data (CPU/memory usage) metric-server try to access the nodes. However, the metric-server cannot resolve the hostname(vps01, vps02, vps03, and vps04) because the ones are not registered in DNS. As you mentioned, you cannot register the hostnames in DNS.

So, you must add the hostnames to /etc/hosts on the node where the metrics-server POD is running.

The autoscaler does not work since the metric-server is not working and there is no metric data.

-- Yoonian
Source: StackOverflow
9/11/2018

I found the following solution:

Change the metrics-server-deployment.yaml file and add:

command:
    - /metrics-server 
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls
-- Lucas Sales
Source: StackOverflow
9/8/2018

It looks like you have DNS issue from your metrics-server pod. You can connect to the pod:

kubectl exec -it metrics-server-xxxxxxxxxx-xxxxx -n kube-system sh
/ # ping vps01

If you can't ping you can't resolve your node.

core-dns or kube-dns use the /etc/resolv.conf on each on your nodes too, so I would check if you can resolve the nodes between each other. Say, can you ping vps01 from vps02 or vps03, etc.

-- Rico
Source: StackOverflow