Kubernetes dashboard doesn't show the consumed CPU and memory by pods

12/13/2018

I successfully connected to the Kubernetes dashboard and I can see all of my deployment, statefulsets, pods, and etc. But the Graphs which determined the amount of CPU and memory using by pods, do not exist.

All the pods :

kube-system   coredns-576cbf47c7-cj8qv                1/1     Running   33         67d
kube-system   coredns-576cbf47c7-qh9hm                1/1     Running   34         67d
kube-system   etcd-master                             1/1     Running   15         67d
kube-system   heapster-684777c4cb-qt6f5               1/1     Running   0          134m
kube-system   kube-apiserver-master                   1/1     Running   23         67d
kube-system   kube-controller-manager-master          1/1     Running   15         67d
kube-system   kube-proxy-bs5k9                        1/1     Running   13         67d
kube-system   kube-proxy-fjp8b                        1/1     Running   13         67d
kube-system   kube-scheduler-master                   1/1     Running   15         67d
kube-system   kubernetes-dashboard-77fd78f978-cnhsc   1/1     Running   0          71m
kube-system   metrics-server-5cbbc84f8c-vz77c         1/1     Running   0          71m
kube-system   monitoring-influxdb-5c5bf4949d-jqr9d    1/1     Running   0          133m
kube-system   weave-net-fl972                         2/2     Running   77         67d
kube-system   weave-net-gh96b                         2/2     Running   34         67d

There are log from dashboard pod :

2018/12/16 08:43:54 Starting overwatch
2018/12/16 08:43:54 Using in-cluster config to connect to apiserver
2018/12/16 08:43:54 Using service account token for csrf signing
2018/12/16 08:43:54 No request provided. Skipping authorization
2018/12/16 08:43:54 Successful initial request to the apiserver, version: v1.12.1
2018/12/16 08:43:54 Generating JWE encryption key
2018/12/16 08:43:54 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/12/16 08:43:54 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/12/16 08:43:55 Initializing JWE encryption key from synchronized object
2018/12/16 08:43:55 Creating in-cluster Heapster client
2018/12/16 08:43:55 Successful request to heapster
2018/12/16 08:43:55 Auto-generating certificates
2018/12/16 08:43:55 Successfully created certificates
2018/12/16 08:43:55 Serving securely on HTTPS port: 8443
2018/12/16 08:44:19 Getting application global configuration
2018/12/16 08:44:19 Application configuration {"serverTime":1544949859551}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/settings/global request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/systembanner request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/rbac/status request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.32.0.1:53200: { contents hidden }
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/overview/default?filterBy=&itemsPerPage=10&name=&page=1&sortBy=d,creationTimestamp request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 Getting config category
2018/12/16 08:44:20 Getting discovery and load balancing category
2018/12/16 08:44:20 Getting lists of all workloads
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 Getting pod metrics
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:24 Getting application global configuration
2018/12/16 08:44:24 Application configuration {"serverTime":1544949864040}

Which says :

Creating in-cluster Heapster client
Successful request to heapster

So the heapster correctly connected to kubernetes dashboard. I use kubeadm version=v1.12.1.

And also when I do kubectl top node in the master node to get the cpu and memory usage it shows the amount of cpu and memory that was used. When I want to access the heapster via teminal like :

curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/

and it returns :

curl: (7) Failed to connect to 10.40.0.63 port 80: Connection refused

The metric-service pod's log :

I1216 12:05:24.783577       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/12/16 12:05:25 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/12/16 12:05:25 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1216 12:05:25.802972       1 serve.go:96] Serving securely on [::]:44

3

Any idea?

-- Majid Rajabi
kubeadm
kubernetes
kubernetes-dashboard

3 Answers

10/23/2019

This is an open issue that is not resolved yet as of this post. https://github.com/kubernetes/dashboard/issues/4145

We are aware, we have a couple of issues with metrics that are not fixed yet. We will work on them.

-- Zstack
Source: StackOverflow

12/14/2018

Quote from README.MD of Kubernetes Dashboard:

Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.

In order to see graphs, you need to deploy grafana.yaml heapster.yaml influxdb.yaml and heapster-rbac.yaml

Then run

kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

You will able to see graphs for pods and deployments.

I hope it will be useful

-- coolinuxoid
Source: StackOverflow

2/26/2019

Heapster has been retired and you may need to modify your heapster installation by following the steps here.

-- Anshul Prakash
Source: StackOverflow