Kubernetes dashboard does not respond when number of Pods are high

10/30/2018

Kubernetes Dashboard works fine before a large number of Pods (more than 1000) are scheduled using a Deployment.

When many Pods are run, I get the following error when trying to click on any page in the Dashboard:
Error: 'dial tcp <ip>:9090: getsockopt: connection refused' Trying to reach: 'http://<ip>:9090/api/v1/workload/default?itemsPerPage=10&page=1'

Does increasing the spec of the master node help?

I am thinking the master node cannot handle the network load?

Update:
might be related, but I'm not sure, when I ran the cluster (I'm using kops) on many medium sized instances instead of a few very large ones, this problem is reduced (I get this error page less).
It seems like the Pod controller on each Node (or whatever you call it) cannot handle so many pods, since a very highly specced Node has many Pods on it whereas a medium sized one will have fewer Pods scheduled on it.

-- cryanbhu
amazon-ec2
kops
kubernetes
kubernetes-dashboard

1 Answer

2/22/2019

It definitely looks like the kubernetes-dashboard pod experienced lack of resources. The possible solution would be to run kubernetes-dashboard pod on the dedicated node or on the node with less resource consumers.

Taints or Affinity features might be helpful to achieve that.

-- VAS
Source: StackOverflow