How to determine the reason why kubernetes node terminated automatically?

7/2/2018

We are running kubernetes in aws and We kept some files in one of the kubernetes nodes(I know it's bad idea to keep data in the host machine instead of separate volume). For some unknown reason, kubernetes node state became NotReady. When I used describe node command, I got status of Memory pressure and Disk pressure are as "NodeStatusUnknown". After some time, that node is terminated and new node is created automatically. Since the old node gets terminated, I couldn't get the data back in the terminated node's root volume. Now, how can I find the reason why it gets terminated?

-- Arun
amazon-web-services
autoscaling
kubernetes

1 Answer

7/2/2018

The node status NotReady causes the health check failed from master node during almost 3 minutes. If you already collect the journal log from the node host, you can investigate why the node service is not responsing. I experienced some NotReady issues, They have various causes, such as networking issue, lack of resources, hang the kubelet service, and so on. First you shoud collect the log to remote centric storage for check.

-- Daein Park
Source: StackOverflow