How to simulate nodeNotReady for a node in Kubernetes

6/20/2019

My ceph cluster is running on AWS with 3 masters 3 workers configuration. When I do kubectl get nodes it shows me all the nodes in the ready state. Is there is any way I can simulate manually to get nodeNotReady error for a node?.

-- Rajat Singh
kubernetes
openshift

3 Answers

7/5/2019

Not sure what is the purpose to simulate NotReady

  1. if the purpose is to not schedule any new pods then you can use kubectl cordon node NODE_NAME This will add the unschedulable taint to it and prevent new pods from being scheduled there.
  2. If the purpose is to evict existing pod then you can use kubectl drain NODE_NAME

In general you can play with taints and toleration to achieve your goal related to the above and you can much more with those!

Now NotReady status comes from the taint node.kubernetes.io/not-ready Ref Which is set by

In version 1.13, the TaintBasedEvictions feature is promoted to beta and enabled by default, hence the taints are automatically added by the NodeController

Therefore if you want to manually set that taint kubectl taint node NODE_NAME node.kubernetes.io/not-ready=:NoExecute the NodeController will reset it automatically!

So to absolutely see the NotReady status this is the best way

Lastly, if you want to remove your networking in a particular node then you can taint it like this kubectl taint node NODE_NAME dedicated/not-ready=:NoExecute

-- garlicFrancium
Source: StackOverflow

6/21/2019

If you just want NodeNotReady you can delete the CNI you have installed. kubectl get all -n kube-system find the DaemonSet of your CNI and delete it or just do a reverse of installing it: kubectl delete -f link_to_your_CNI_yaml

You could also try to overwhelm the node with too many pods (resources). You can also share your main goal so we can adjust the answer.

About the answer from P Ekambaram you could just ssh to a node and then stop the kubelet.

To do that in kops you can just:

ssh -A admin@Node_PublicDNS_name

systemctl stop kubelet

EDIT: Another way is to overload the Node which will cause: System OOM encountered and that will result in Node NotReady state. This is just one of the ways of how to achieve it: SSH into the Node you want to get into NotReady Install Stress Run stress: stress --cpu 8 --io 4 --hdd 10 --vm 4 --vm-bytes 1024M --timeout 5m (you can adjust the values of course) Wait till Node crash. After you stop the stress the Node should get back to healthy state automatically.

-- aurelius
Source: StackOverflow

6/20/2019

just stop kebelet service on one of the node that you want to see as NodeNotReady

-- P Ekambaram
Source: StackOverflow