Kubernetes node NotReady perhaps due to invalid memory address or nil pointer dereference in reconciler.go

2/2/2019

My experimental, two-node Kubernetes 1.13.2 unfortunately entered a mode where the second node is NotReady. I've tried systemctl restart kubelet on both nodes, but this has not helped so far.

journalctl -u kubelet.service on the master node ends with this line:

reconciler.go:154] Reconciler: start to sync state

journalctl -u kubectl.service on the second node contains these lines:

server.go:999] Started kubelet
kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
kubelet.go:1412] No api server defined - no node status update will be sent.
fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
status_manager.go:148] Kubernetes client is nil, not starting status manager.
kubelet.go:1829] Starting kubelet main sync loop.
kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet PLEG is not healthy: pleg has yet to be successful]
server.go:137] Starting to listen on 0.0.0.0:10250
server.go:333] Adding debug handlers to kubelet server.
volume_manager.go:248] Starting Kubelet Volume Manager
desired_state_of_world_populator.go:130] Desired state populator starts to run
container.go:409] Failed to create summary reader for "/system.slice/atop.service": none of the resources are being tracked.
kubelet.go:1846] skipping pod synchronization - [container runtime status check may not have completed yet]
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
cpu_manager.go:155] [cpumanager] starting with none policy
cpu_manager.go:156] [cpumanager] reconciling every 10s
policy_none.go:42] [cpumanager] none policy: Start
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
reconciler.go:154] Reconciler: start to sync state
runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:562
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:599
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:419
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:330
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:155
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/workspace/anago-v1.13.2-beta.0.75+cff46ab41ff0bb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:143
/usr/local/go/src/runtime/asm_amd64.s:1333
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
...

What's going on here and how can I remedy this situation?

UPDATE I've now stopped kubelet.service on both nodes and then removed all docker containers (docker rm) and images (docker rmi) also on both nodes. I've then restarted kubelet.service on the master only. Kubernetes apparently pulls in all of its own images and starts them again, but now see multiple connection refused errors in journalctl -u kubelet.service, such as:

kubelet_node_status.go:94] Unable to register node "my_node" with API server: Post https://my_master:6443/api/v1/nodes: dial tcp my_master:6443: connect: connection refused

kubectl get nodes can still access the API server normally. So perhaps this closer to the root cause of what I am facing. Is it correct to assume that at this point the kubelet is trying to connect to the API server? How can I check whether the relevant credentials are still intact?

-- rookie099
kubernetes

2 Answers

3/18/2019

The issue remained unexplained and I had to recreate the cluster with kubeadm reset, etc.

-- rookie099
Source: StackOverflow

2/2/2019

It is kinda hard to troubleshoot without seeing the whole configuration.

however try to follow this check list.

-- cperez08
Source: StackOverflow