I've recently noticed that a GKE node in state NodeNotSchedulable
is removed from the list of nodes in the HTTP load balancer forwarding rules.
Is there a way to disable this?
It's not a wanted behavior, because if you cordon nodes which contain nginx pods, for instance, they will be no longer reachable.
Thanks.
This happens because when you "cordon" a node, you're probably also about to run a "drain" to evict all the pods on the node. (Assumption is that cordon+drain are used together.)
Therefore, prior to an eviction, cordoned nodes deliberately return failure to load balancing health checks, so that the pods they host are slowly taken out of Load Balancing list. That's why you're seeing the unhealthy signals.
This ensures that your cordoned node stops getting new traffic, therefore the pods on it can be evicted (=drained) more gracefully since they won't be handling active requests.
(I am failing to find the documentation on this but I've definitely read it somewhere. If I can find, I'll attach it here).
I recommend you to use other/custom "taints" to cordon your nodes.