My experimental, two-node Kubernetes 1.13.2 unfortunately entered a mode where the second node is NotReady
. I've tried systemctl restart kubelet
on both nodes, but this has not helped so far.
journalctl -u kubelet.service
on the master node ends with this line:
reconciler.go:154] Reconciler: start to sync state
journalctl -u kubectl.service
on the second node contains these lines:
server.go:999] Started kubelet
kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
kubelet.go:1412] No api server defined - no node status update will be sent.
fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
status_manager.go:148] Kubernetes client is nil, not starting status manager.
kubelet.go:1829] Starting kubelet main sync loop.
kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful]
server.go:137] Starting to listen on 0.0.0.0:10250
server.go:333] Adding debug handlers to kubelet server.
volume_manager.go:248] Starting Kubelet Volume Manager
desired_state_of_world_populator.go:130] Desired state populator starts to run
container.go:409] Failed to create summary reader for "/system.slice/atop.service": none of the resources are being tracked.
kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet]
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
cpu_manager.go:155] [cpumanager] starting with none policy
cpu_manager.go:156] [cpumanager] reconciling every 10s
policy_none.go:42] [cpumanager] none policy: Start
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
reconciler.go:154] Reconciler: start to sync state
runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:562
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:599
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:419
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:330
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:155
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:143
/usr/local/go/src/runtime/asm_amd64.s:1333
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
...
What's going on here and how can I remedy this situation?
UPDATE I've now stopped kubelet.service
on both nodes and then removed all docker containers (docker rm
) and images (docker rmi
) also on both nodes. I've then restarted kubelet.service
on the master only. Kubernetes apparently pulls in all of its own images and starts them again, but now see multiple connection refused errors in journalctl -u kubelet.service
, such as:
kubelet_node_status.go:94] Unable to register node "my_node" with API server: Post https://my_master:6443/api/v1/nodes: dial tcp my_master:6443: connect: connection refused
kubectl get nodes
can still access the API server normally. So perhaps this closer to the root cause of what I am facing. Is it correct to assume that at this point the kubelet is trying to connect to the API server? How can I check whether the relevant credentials are still intact?
The issue remained unexplained and I had to recreate the cluster with kubeadm reset
, etc.
It is kinda hard to troubleshoot without seeing the whole configuration.
however try to follow this check list.