Weavescope dashboard displays No data from cluster (empty dashboard after successful install)
1/16/2020
I have successfully deployed weavscope on my local cluster, the weavscope dashboard is reachable. Issue is it dashboard displays no data but just a blank screen like below (tested all tabs and view)
I have used this weavscope.yaml to deploy
Steps to install and port forward I have ensured the IP used and commands have no syntax error.
$ wget https://cloud.weave.works/launch/k8s/weavescope.yaml
--2020-01-16 12:54:52-- https://cloud.weave.works/launch/k8s/weavescope.yaml
Resolving cloud.weave.works (cloud.weave.works)... 52.200.96.42, 34.231.197.104
Connecting to cloud.weave.works (cloud.weave.works)|52.200.96.42|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /k8s/v1.13/scope.yaml [following]
--2020-01-16 12:54:53-- https://cloud.weave.works/k8s/v1.13/scope.yaml
Reusing existing connection to cloud.weave.works:443.
HTTP request sent, awaiting response... 200 OK
Length: 9964 (9.7K) [application/x-yaml]
Saving to: ‘weavescope.yaml’
weavescope.yaml 100%[====================================================================================================================>] 9.73K --.-KB/s in 0s
2020-01-16 12:54:53 (243 MB/s) - ‘weavescope.yaml’ saved [9964/9964]
$ ll
total 20
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 16 12:54 ./
drwxr-xr-x 8 ubuntu ubuntu 4096 Jan 16 12:54 ../
-rw-rw-r-- 1 ubuntu ubuntu 9964 Jan 16 12:54 weavescope.yaml
$ kubectl cluster-info
Kubernetes master is running at https://xx.xx.xx.xx:6443
KubeDNS is running at https://xx.xx.xx.xx:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl create -f weavescope.yaml
namespace/weave created
serviceaccount/weave-scope created
clusterrole.rbac.authorization.k8s.io/weave-scope created
clusterrolebinding.rbac.authorization.k8s.io/weave-scope created
deployment.apps/weave-scope-app created
service/weave-scope-app created
deployment.apps/weave-scope-cluster-agent created
daemonset.apps/weave-scope-agent created
$ pod=$(kubectl get pod -n weave --selector=name=weave-scope-app -o jsonpath={.items..metadata.name})
$ kubectl expose pod $pod -n weave --external-ip="xx.xx.xx.xx" --port=4040 --target-port=4040Its stuck in fetching pods state not able to figure out what is blocking the data fetch when i can reach the dashboard on given port after port forward
I have checked there are pods running on my cluster and no issue on cluster side can spin new pods etc ..
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/busybox-c44c795b6-c424g 1/1 Running 0 16s
pod/nginx-86c57db685-znqh2 1/1 Running 0 3d1h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d4h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/busybox 1/1 1 1 16s
deployment.apps/nginx 1/1 1 1 3d1h
NAME DESIRED CURRENT READY AGE
replicaset.apps/busybox-c44c795b6 1 1 1 16s
replicaset.apps/nginx-86c57db685 1 1 1 3d1h
**More Logs**
$ kubectl get pods -n weave
NAME READY STATUS RESTARTS AGE
weave-scope-agent-cshjf 1/1 Running 0 59m
weave-scope-agent-ddcjv 1/1 Running 0 59m
weave-scope-agent-krt6t 1/1 Running 0 59m
weave-scope-agent-pvjcg 1/1 Running 0 59m
weave-scope-app-848cd4d8b5-22qhv 1/1 Running 0 59m
weave-scope-cluster-agent-b4f45797c-sxtfb 1/1 Running 0 59m
$ kubectl -n weave logs weave-scope-app-848cd4d8b5-22qhv
<app> INFO: 2020/01/16 12:55:42.641840 app starting, version 1.12.0, ID 27b9ceab0b49f8d7
<app> INFO: 2020/01/16 12:55:42.641879 command line args: --mode=app
<app> INFO: 2020/01/16 12:55:42.643881 Basic authentication disabled
<app> INFO: 2020/01/16 12:55:42.645552 listening on :4040
$ kubectl -n weave logs weave-scope-agent-cshjf
time="2020-01-16T12:55:33Z" level=info msg="publishing to: weave-scope-app.weave.svc.cluster.local:80"
<probe> INFO: 2020/01/16 12:55:33.746901 Basic authentication disabled
<probe> INFO: 2020/01/16 12:56:13.749823 command line args: --mode=probe --probe-only=true --probe.docker=true --probe.docker.bridge=docker0 --probe.kubernetes.role=host --probe.publish.interval=4.5s --probe.spy.interval=2s weave-scope-app.weave.svc.cluster.local:80
<probe> INFO: 2020/01/16 12:56:13.749871 probe starting, version 1.12.0, ID 1a14cbb93863dc96
<probe> WARN: 2020/01/16 12:56:13.751629 Cannot resolve 'scope.weave.local.': dial tcp 172.17.0.1:53: connect: connection refused
<probe> WARN: 2020/01/16 12:56:14.144400 Error collecting weave status, backing off 10s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:24.144816 Error collecting weave status, backing off 20s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> ERRO: 2020/01/16 12:56:43.814003 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kernel-version=4.4.0-169-generic&flag_kubernetes_enabled=true&flag_os=linux&os=linux&signature=2irqD0pagYmnL6vzwsw6tNzZ2ASN1WIgEfUhtKbaJNM%3D&version=1.12.0: dial tcp: i/o timeout
<probe> WARN: 2020/01/16 12:56:44.145328 Error collecting weave status, backing off 40s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:53.752597 Cannot resolve 'weave-scope-app.weave.svc.cluster.local': lookup weave-scope-app.weave.svc.cluster.local on 10.96.0.10:53: read udp 131.160.200.108:42692->10.96.0.10:53: i/o timeout
<probe> ERRO: 2020/01/16 12:57:13.904958 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kernel-version=4.4.0-169-generic&flag_kubernetes_enabled=true&flag_os=linux&os=linux&signature=2irqD0pagYmnL6vzwsw6tNzZ2ASN1WIgEfUhtKbaJNM%3D&version=1.12.0: dial tcp: i/o timeout
$ kubectl -n weave logs weave-scope-agent-ddcjv
time="2020-01-16T12:55:26Z" level=info msg="publishing to: weave-scope-app.weave.svc.cluster.local:80"
<probe> INFO: 2020/01/16 12:55:26.417234 Basic authentication disabled
<probe> INFO: 2020/01/16 12:56:06.420120 command line args: --mode=probe --probe-only=true --probe.docker=true --probe.docker.bridge=docker0 --probe.kubernetes.role=host --probe.publish.interval=4.5s --probe.spy.interval=2s weave-scope-app.weave.svc.cluster.local:80
<probe> INFO: 2020/01/16 12:56:06.420160 probe starting, version 1.12.0, ID 5bee7195c53ead34
<probe> WARN: 2020/01/16 12:56:06.422164 Cannot resolve 'scope.weave.local.': dial tcp 172.17.0.1:53: connect: connection refused
<probe> WARN: 2020/01/16 12:56:06.810258 Error collecting weave status, backing off 10s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:16.810782 Error collecting weave status, backing off 20s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> ERRO: 2020/01/16 12:56:36.500739 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kernel-version=4.4.0-169-generic&flag_kubernetes_enabled=true&flag_os=linux&os=linux&signature=4MV4m5TneqzOVfI9lTdXt8Lmjj0RBnU5u85Dx%2BO4cZ4%3D&version=1.12.0: dial tcp: i/o timeout
<probe> WARN: 2020/01/16 12:56:36.811255 Error collecting weave status, backing off 40s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:46.423041 Cannot resolve 'weave-scope-app.weave.svc.cluster.local': lookup weave-scope-app.weave.svc.cluster.local on 10.96.0.10:53: read udp 131.160.200.109:52151->10.96.0.10:53: i/o timeout
<probe> ERRO: 2020/01/16 12:57:06.564171 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kernel-version=4.4.0-169-generic&flag_kubernetes_enabled=true&flag_os=linux&os=linux&signature=4MV4m5TneqzOVfI9lTdXt8Lmjj0RBnU5u85Dx%2BO4cZ4%3D&version=1.12.0: dial tcp: i/o timeout
$ kubectl -n weave logs weave-scope-agent-krt6t
time="2020-01-16T12:55:20Z" level=info msg="publishing to: weave-scope-app.weave.svc.cluster.local:80"
<probe> INFO: 2020/01/16 12:55:20.078069 Basic authentication disabled
<probe> INFO: 2020/01/16 12:55:20.081051 command line args: --mode=probe --probe-only=true --probe.docker=true --probe.docker.bridge=docker0 --probe.kubernetes.role=host --probe.publish.interval=4.5s --probe.spy.interval=2s weave-scope-app.weave.svc.cluster.local:80
<probe> INFO: 2020/01/16 12:55:20.081100 probe starting, version 1.12.0, ID 30049d92f78e0dd8
<probe> WARN: 2020/01/16 12:55:20.082543 Cannot resolve 'scope.weave.local.': dial tcp 172.17.0.1:53: connect: connection refused
<probe> WARN: 2020/01/16 12:55:20.427313 Error collecting weave status, backing off 10s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> ERRO: 2020/01/16 12:55:21.084863 Error fetching app details: Get http://10.96.86.6:80/api: dial tcp 10.96.86.6:80: connect: connection refused
<probe> ERRO: 2020/01/16 12:55:23.084173 Error fetching app details: Get http://10.96.86.6:80/api: dial tcp 10.96.86.6:80: connect: connection refused
<probe> ERRO: 2020/01/16 12:55:25.084261 Error fetching app details: Get http://10.96.86.6:80/api: dial tcp 10.96.86.6:80: connect: connection refused
<probe> ERRO: 2020/01/16 12:55:29.084329 Error fetching app details: Get http://10.96.86.6:80/api: dial tcp 10.96.86.6:80: connect: connection refused
<probe> WARN: 2020/01/16 12:55:30.427831 Error collecting weave status, backing off 20s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> ERRO: 2020/01/16 12:55:37.084216 Error fetching app details: Get http://10.96.86.6:80/api: dial tcp 10.96.86.6:80: connect: connection refused
<probe> INFO: 2020/01/16 12:55:49.089567 Control connection to weave-scope-app.weave.svc.cluster.local starting
<probe> WARN: 2020/01/16 12:55:50.428409 Error collecting weave status, backing off 40s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> INFO: 2020/01/16 12:55:51.938293 Publish loop for weave-scope-app.weave.svc.cluster.local starting
$ kubectl -n weave logs weave-scope-agent-pvjcg
time="2020-01-16T12:55:33Z" level=info msg="publishing to: weave-scope-app.weave.svc.cluster.local:80"
<probe> INFO: 2020/01/16 12:55:33.880079 Basic authentication disabled
<probe> INFO: 2020/01/16 12:56:13.883365 command line args: --mode=probe --probe-only=true --probe.docker=true --probe.docker.bridge=docker0 --probe.kubernetes.role=host --probe.publish.interval=4.5s --probe.spy.interval=2s weave-scope-app.weave.svc.cluster.local:80
<probe> INFO: 2020/01/16 12:56:13.883410 probe starting, version 1.12.0, ID 4c0cce1880a52ef8
<probe> WARN: 2020/01/16 12:56:13.884834 Cannot resolve 'scope.weave.local.': dial tcp 172.17.0.1:53: connect: connection refused
<probe> WARN: 2020/01/16 12:56:14.240728 Error collecting weave status, backing off 10s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:24.241188 Error collecting weave status, backing off 20s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> ERRO: 2020/01/16 12:56:43.949173 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kernel-version=4.4.0-169-generic&flag_kubernetes_enabled=true&flag_os=linux&os=linux&signature=e%2FbJ2Qy32Xq3WG0vkozjYngRMp%2B028dfFox%2F9vTLd%2BQ%3D&version=1.12.0: dial tcp: i/o timeout
<probe> WARN: 2020/01/16 12:56:44.241645 Error collecting weave status, backing off 40s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:53.885827 Cannot resolve 'weave-scope-app.weave.svc.cluster.local': lookup weave-scope-app.weave.svc.cluster.local on 10.96.0.10:53: read udp 131.160.200.110:54514->10.96.0.10:53: i/o timeout
<probe> ERRO: 2020/01/16 12:57:14.014110 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kernel-version=4.4.0-169-generic&flag_kubernetes_enabled=true&flag_os=linux&os=linux&signature=e%2FbJ2Qy32Xq3WG0vkozjYngRMp%2B028dfFox%2F9vTLd%2BQ%3D&version=1.12.0: dial tcp: i/o timeout
$ kubectl -n weave logs weave-scope-cluster-agent-b4f45797c-sxtfb
time="2020-01-16T12:55:38Z" level=info msg="publishing to: weave-scope-app.weave.svc.cluster.local:80"
<probe> INFO: 2020/01/16 12:55:38.759547 Basic authentication disabled
<probe> INFO: 2020/01/16 12:55:38.763837 command line args: --mode=probe --probe-only=true --probe.http.listen=:4041 --probe.kubernetes.role=cluster --probe.publish.interval=4.5s --probe.spy.interval=2s weave-scope-app.weave.svc.cluster.local:80
<probe> INFO: 2020/01/16 12:55:38.763869 probe starting, version 1.12.0, ID 41240071be33a064
<probe> ERRO: 2020/01/16 12:55:38.763946 Error getting docker bridge ip: route ip+net: no such network interface
<probe> INFO: 2020/01/16 12:55:38.765521 kubernetes: targeting api server https://10.96.0.1:443
<probe> ERRO: 2020/01/16 12:55:38.768123 plugins: problem loading: no such file or directory
<probe> INFO: 2020/01/16 12:55:38.769589 Profiling data being exported to :4041
<probe> INFO: 2020/01/16 12:55:38.769606 go tool pprof http://:4041/debug/pprof/{profile,heap,block}
<probe> WARN: 2020/01/16 12:55:38.770504 Error collecting weave status, backing off 10s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> INFO: 2020/01/16 12:55:38.785340 volumesnapshots are not supported by this Kubernetes version
<probe> INFO: 2020/01/16 12:55:38.790283 volumesnapshotdatas are not supported by this Kubernetes version
<probe> INFO: 2020/01/16 12:55:39.774173 Control connection to weave-scope-app.weave.svc.cluster.local starting
<probe> INFO: 2020/01/16 12:55:43.271912 Publish loop for weave-scope-app.weave.svc.cluster.local starting
<probe> WARN: 2020/01/16 12:55:48.771005 Error collecting weave status, backing off 20s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.
<probe> WARN: 2020/01/16 12:56:08.771352 Error collecting weave status, backing off 40s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: connect: connection refused. If you are not running Weave Net, you may wish to suppress this warning by launching scope with the `--weave=false` option.-- DT.
kubectl
kubernetes
Similar Questions
1 Answer
1/20/2020
I am not sure if i solved it correctly or not but i got the data working with below changes on deployment.
To start with the error logs seen on "weave-scope-agent" "weave-scope-app" and turned out to be red herring and after spending lot of time there i turned back to my cluster install to check if i had missed (messed-up) something at install time. Doing so i identified on cluster Master node i had all port open but my worker nodes did not (i.e a restrictive security group was in use) Just that when i created the kubernetes cluster few days back while creating ubuntu vm i had applied allow all rule to the machine where i was going to setup master but chose a restrictive security group rules for the workers nodes.
So as a debug test decided to apply same security group rules across my worker nodes as well to match the rules used on master node. After having same rules across all nodes on kubernetes cluster (master + worker) was able to get the dashboard loading the data. So can conclude this issue as a deployment time issue on my side.
Data is correctly loaded on dashboard now with cluster security group rules corrected across all nodes.
Since the security rules are on my private network i will shared just one common example from what i added for any other users looking for reference if they face such issue in future
Example :
Ingress IPv4 TCP 30000 - 32767 0.0.0.0/0-- DT.
Source: StackOverflow