How to watch nodes and get notifies if they become "NotReady" using client-go or curl?

12/27/2017

If some nodes down in a cluster and the status become NotReady, can I make it to trigger some action?

I tried to use client-go and curl to watch /api/v1/watch/nodes and get notifies when nodes status have any change. How do I get notifies only if nodes status change from Ready to NotReady?

-- Jacky Hung
curl
kubernetes

1 Answer

12/27/2017

As of now, there is no way to query for specific updates. There are two ways you can implement this:

  1. The NodeStatus has a field called lastTransitionTime which records a change in a field. In your application, you can check every X seconds and then compare last checked time with lastTransitionTime to determine if there is a change. If your application restarts, you will have to do a one time check at startup of application

    {
            "type": "Ready",
            "status": "True",
            "lastHeartbeatTime": "2017-12-27T09:52:19Z",
            "lastTransitionTime": "2017-12-26T14:55:49Z",
            "reason": "KubeletReady",
            "message": "kubelet is posting ready status"
    }
    
  2. Your app written in client-go can maintain a cache and compare the value in cache to the value from watch events. In this case as well, you will have to think of ways to not miss events when your watcher service is down. You can either store that state somewhere or use some other mechanism but it depends on criticality of your use case.

The closest example I can find is in Calico-go, where any changes in network policies are watched for and related changes are done in a calico-etcd so that agents on nodes can implement firewall rules accordingly. Starting on this line, events are handled, of course you will have to navigate the stack further! Code

-- Vishal Biyani
Source: StackOverflow