Why does HTTP load balancer forwarding rules exclude cordoned GKE nodes?

5/23/2018

I've recently noticed that a GKE node in state NodeNotSchedulable is removed from the list of nodes in the HTTP load balancer forwarding rules.

Is there a way to disable this?

It's not a wanted behavior, because if you cordon nodes which contain nginx pods, for instance, they will be no longer reachable.

Thanks.

Load Balancer forwarding rules

-- erkez
google-cloud-platform
google-kubernetes-engine

1 Answer

5/24/2018

This happens because when you "cordon" a node, you're probably also about to run a "drain" to evict all the pods on the node. (Assumption is that cordon+drain are used together.)

Therefore, prior to an eviction, cordoned nodes deliberately return failure to load balancing health checks, so that the pods they host are slowly taken out of Load Balancing list. That's why you're seeing the unhealthy signals.

This ensures that your cordoned node stops getting new traffic, therefore the pods on it can be evicted (=drained) more gracefully since they won't be handling active requests.

(I am failing to find the documentation on this but I've definitely read it somewhere. If I can find, I'll attach it here).

I recommend you to use other/custom "taints" to cordon your nodes.

-- AhmetB - Google
Source: StackOverflow