K
Q

What makes a kubernetes node unhealthy?

9/13/2018

We've experienced 4 AUTO_REPAIR_NODES events(revealed by the command gcloud container operations list) on our GKE cluster during the past 1 month. The consequence of node-auto-repair is that the node gets recreated and gets attached a new external IP, and the new external IP, which was not whitelisted by third-party services, eventually caused failure of services running on that the new node.

I noticed that we have "Automatic node repair" enabled in our Kubernetes cluster and felt tempted to disable that, but before I do that, I need to know more about the situation.

My questions are:

  1. What are some common causes that makes a node unhealthy in the first place? I'm aware of this article https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair#node_repair_process which says, "a node reports a NotReady status on consecutive checks over the given time threshold" would trigger auto repair. But what could cause a node to become NotReady?
  2. I'm also aware of this article https://kubernetes.io/docs/concepts/architecture/nodes/#node-status which mentions the full list of node status: {OutOfDisk, Ready, MemoryPressure, PIDPressure, DiskPressure, NetworkUnavailable, ConfigOK}. I wonder, if any of {OutOfDisk, MemoryPressure, PIDPressure, DiskPressure, NetworkUnavailable} becomes true for a node, would that node becomes NotReady?
  3. What negative consequences could I get after I disable "Automatic node repair" in the cluster? I'm basically wondering whether we could end up in a worse situation than auto-repaired nodes and newly-attached-not-whitelisted IP. Once "Automatic node repair" is disabled, then for the pods that are running on an Unhealthy node that would've been auto-repaired, would Kubernetes create new pods on other nodes?
-- twimo
google-cloud-platform
google-kubernetes-engine
kubernetes

Similar Questions

kubernetes deployment hyperledger fabric
How can I do this to approve a CSR in the Denied state?
Adding Custom Header to Logout in Icecast
Permission denied while deploying/activating docker image in Rancher-Kubernetes
How to handle concurrent transactions from multiple pods in kubernetes
Installing Jenkins onminikube shows Failed to pull image "jenkins/jenkins:2.303.3-jdk11"
Traefik Kubernetes 301 Redirect

1 Answer

9/13/2018

The confusion lies here in that there are 'Ready' and 'NotReady' states that are shown when you run kubectl get nodes which are reported by the kube-apiserver. But these are independent and unclear from the docs how they relate to the kubelet states described here You can also see the kubelet states (in events) when you run kubectl describe nodes

To answer some parts of the questions:

  1. As reported by the kube-apiserver

    • Kubelet down
    • docker or containerd or crio down (depending on the shim you are using)
    • kubelet states - unclear.

  2. For these, the kubelet will start evicting or not scheduling pods except for Ready (https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/). Unclear from the docs how these get reported from the kubeapi-server.

    • You could have nodes on your cluster not being used and you'd be paying for that usage.
    • Yes, k8s will reschedule the pods after a certain readiness probes fail (configurable). If the kubelet is down or the node down k8s will think the pods are down.
    • Assuming your nodes go down, you could end up with less capacity than what you need to schedule your workloads to k8s would not be able to schedule them anyway.

Hope it helps!

-- Rico
Source: StackOverflow