K
Q

How to troubleshoot metrics-server on kubeadm?

7/21/2019

I have a working 1.15.1 kubenetes cluster using kubeadm on bare-metal and just deployed metrics-server as in the docs:

git clone https://github.com/kubernetes-incubator/metrics-server.git
kubectl create -f metrics-server/deploy/1.8+/

After some time I try kubectl top node and I get as response:

error: metrics not available yet

Also when I try kubectl top pods I get:

W0721 20:01:31.786615 21232 top_pod.go:266] Metrics not available for pod default/pod-deployment-57b99df6b4-khh84, age: 27h31m59.78660593s error: Metrics not available for pod default/pod-deployment-57b99df6b4-khh84, age: 27h31m59.78660593s

I checked the pod and service for metrics-server and all of them are running fine. Where should I try to see a problem?

-- staticdev
kubeadm
kubernetes
monitoring

Similar Questions

How to fix environment variables not being picked up in a .json file
Running multiple instances of same springboot application
Can someone explain me some use cases of helm?
Nightly TF / Cloned TFX - how to manage Image for Kubeflow?
Python Flask weird logging behavior (kubernetes)
kubernetes k3s agent can't connect to master CA/SSL error

2 Answers

7/25/2019

Edit the metric-server deployment like Subramanian Manickam's answer said, you can also do it with

$ kubectl edit deploy -n kube-system metrics-server

That will open a text editor with the deployment yaml-file where you can make the following changes:

Under spec.template.spec.containers, on the same level as name: metrics-server add

args:
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --metric-resolution=30s

and then under spec.template.spec at the same level as containers I also had to add:

hostNetwork: true

to fix the metrics-server working with the CNI (calico in my case).

Afterwards your deployment yaml should look something like this:

[...]
spec:
  [...]
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: metrics-server
      name: metrics-server
    spec:
      containers:
      - args:
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-insecure-tls
        - --metric-resolution=30s
        image: k8s.gcr.io/metrics-server-amd64:v0.3.3
        imagePullPolicy: Always
        name: metrics-server
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      dnsPolicy: ClusterFirst
      hostNetwork: true
[...]

After that it took about 10-15s for kubectl top pods to return some data.

-- char
Source: StackOverflow
7/22/2019

You have to add this command section after line number #33 on metrics-server-deployment.yaml file.

  command:
    - /metrics-server
    - --kubelet-preferred-address-types=InternalIP
    - --kubelet-insecure-tls

Once you have updated the file, you have to re-deploy the pod.

-- Subramanian Manickam
Source: StackOverflow