How to debug kuberenetes system pod elections on masters?

6/30/2017

I'm trying to make SSL work with kubernetes, but am stuck with leader election problem. So I think I should be seeing scheduler and controller system pods somewhere, while all I have is this:

kubectl get po --namespace=kube-system NAME READY STATUS RESTARTS AGE kube-apiserver-10.255.12.200 1/1 Running 0 18h kube-apiserver-10.255.16.111 1/1 Running 0 20h kube-apiserver-10.255.17.12 1/1 Running 0 20h scheduler-master-10.255.12.200 2/2 Running 0 20h scheduler-master-10.255.16.111 2/2 Running 0 20h scheduler-master-10.255.17.12 2/2 Running 0 20h

as for comparison, on my other clusters, I can see more pods:

kubectl get po --namespace=kube-system NAME READY STATUS RESTARTS AGE kube-apiserver-10.255.0.248 1/1 Running 1 30d kube-apiserver-10.255.1.112 1/1 Running 1 30d kube-apiserver-10.255.1.216 1/1 Running 1 30d kube-controller-manager-10.255.1.216 1/1 Running 3 30d kube-scheduler-10.255.1.216 1/1 Running 1 30d scheduler-master-10.255.0.248 2/2 Running 2 30d scheduler-master-10.255.1.112 2/2 Running 2 30d scheduler-master-10.255.1.216 2/2 Running 2 30d

Does anybody knows how to debug this ? Pod logs doesn't show much, and my pods are stuck in pending state.

-- sokoow
docker
kubernetes

2 Answers

7/4/2017

Several things could be happening:

  • Were the two clusters created the same way? Note that, depending on the cluster, some services like the scheduler or the controller could be running on bare metal. If that is the case, then you should check the logs inside the node using, for instance, systemctl status <name-of-the-service>.service (in case it is using systemd).

  • In case these modules are meant to be running as pods, then I would advise you to go to the master node and check /etc/kubernetes/manifests. You should find the maniphests of the services you are looking for. If they are not, then you have the reason why you do not find the pods in the system. If the pods are there, then check if the arguments are properly set (especially the --leader-elect one)

-- Javier Salmeron
Source: StackOverflow

7/4/2017

As mentioned by Javier Salmeron check the manifest folder. If they do exist there then check the log output of kubelet. If the missing pods try to start yout could also check the logs with "docker logs"

-- Milo van der Zee
Source: StackOverflow