I successfully connected to the Kubernetes dashboard and I can see all of my deployment
, statefulsets
, pods
, and etc. But the Graphs which determined the amount of CPU and memory using by pods, do not exist.
All the pods :
kube-system coredns-576cbf47c7-cj8qv 1/1 Running 33 67d
kube-system coredns-576cbf47c7-qh9hm 1/1 Running 34 67d
kube-system etcd-master 1/1 Running 15 67d
kube-system heapster-684777c4cb-qt6f5 1/1 Running 0 134m
kube-system kube-apiserver-master 1/1 Running 23 67d
kube-system kube-controller-manager-master 1/1 Running 15 67d
kube-system kube-proxy-bs5k9 1/1 Running 13 67d
kube-system kube-proxy-fjp8b 1/1 Running 13 67d
kube-system kube-scheduler-master 1/1 Running 15 67d
kube-system kubernetes-dashboard-77fd78f978-cnhsc 1/1 Running 0 71m
kube-system metrics-server-5cbbc84f8c-vz77c 1/1 Running 0 71m
kube-system monitoring-influxdb-5c5bf4949d-jqr9d 1/1 Running 0 133m
kube-system weave-net-fl972 2/2 Running 77 67d
kube-system weave-net-gh96b 2/2 Running 34 67d
There are log from dashboard pod :
2018/12/16 08:43:54 Starting overwatch
2018/12/16 08:43:54 Using in-cluster config to connect to apiserver
2018/12/16 08:43:54 Using service account token for csrf signing
2018/12/16 08:43:54 No request provided. Skipping authorization
2018/12/16 08:43:54 Successful initial request to the apiserver, version: v1.12.1
2018/12/16 08:43:54 Generating JWE encryption key
2018/12/16 08:43:54 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2018/12/16 08:43:54 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2018/12/16 08:43:55 Initializing JWE encryption key from synchronized object
2018/12/16 08:43:55 Creating in-cluster Heapster client
2018/12/16 08:43:55 Successful request to heapster
2018/12/16 08:43:55 Auto-generating certificates
2018/12/16 08:43:55 Successfully created certificates
2018/12/16 08:43:55 Serving securely on HTTPS port: 8443
2018/12/16 08:44:19 Getting application global configuration
2018/12/16 08:44:19 Application configuration {"serverTime":1544949859551}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/settings/global request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/systembanner request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/rbac/status request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.32.0.1:53200: { contents hidden }
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Incoming HTTP/2.0 GET /api/v1/overview/default?filterBy=&itemsPerPage=10&name=&page=1&sortBy=d,creationTimestamp request from 10.32.0.1:53200: {}
2018/12/16 08:44:20 Getting config category
2018/12/16 08:44:20 Getting discovery and load balancing category
2018/12/16 08:44:20 Getting lists of all workloads
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 No metric client provided. Skipping metrics.
2018/12/16 08:44:20 Getting pod metrics
2018/12/16 08:44:20 [2018-12-16T08:44:20Z] Outcoming response to 10.32.0.1:53200 with 200 status code
2018/12/16 08:44:24 Getting application global configuration
2018/12/16 08:44:24 Application configuration {"serverTime":1544949864040}
Which says :
Creating in-cluster Heapster client
Successful request to heapster
So the heapster correctly connected to kubernetes dashboard. I use kubeadm version=v1.12.1
.
And also when I do kubectl top node
in the master node to get the cpu and memory usage it shows the amount of cpu and memory that was used. When I want to access the heapster via teminal like :
curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/
and it returns :
curl: (7) Failed to connect to 10.40.0.63 port 80: Connection refused
The metric-service pod's log :
I1216 12:05:24.783577 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/12/16 12:05:25 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/12/16 12:05:25 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1216 12:05:25.802972 1 serve.go:96] Serving securely on [::]:44
3
Any idea?
This is an open issue that is not resolved yet as of this post. https://github.com/kubernetes/dashboard/issues/4145
We are aware, we have a couple of issues with metrics that are not fixed yet. We will work on them.
Quote from README.MD of Kubernetes Dashboard:
Heapster has to be running in the cluster for the metrics and graphs to be available. Read more about it in Integrations guide.
In order to see graphs, you need to deploy grafana.yaml heapster.yaml influxdb.yaml and heapster-rbac.yaml
Then run
kubectl delete -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
You will able to see graphs for pods and deployments.
I hope it will be useful
Heapster has been retired and you may need to modify your heapster installation by following the steps here.