Proper way to elect a leader from a set of candidate pods in Kubernetes

7/1/2019

To perform the leader election, Kubernetes documentation suggests to deploy a side car in the set of candidate pods.

https://kubernetes.io/blog/2016/01/simple-leader-election-with-kubernetes/

This side car follows following steps to elect a leader.

  1. If a valid leader does not exist, every side cars tries to update a Kubernetes endpoint object atomically. Only one of the side car can successfully update the endpoint object.
  2. The side car which updates the endpoint will assume as the leader for a specified time duration.
  3. Current leader will update the endpoint again to extend the time duration in order to retain the leadership.
  4. Other side cars will not try to update the endpoint again when a valid leader exists.
  5. If the current leader does not update the endpoint within the time duration, other side cars will consider that the leadership is revoked. All side cars will go to step 1.

There are few issues with this method.

  1. It is possible to run 2 leaders simultaneously for a brief period of time. Example:

If the current leader hangs and cannot update the endpoint in time, one of the other side cars will acquire the leadership. But it will take some time for the previous leader to realize that its leadership status is revoked. For this brief period of time, existing 2 leaders can corrupt a shared resource/data.

This is also mentioned in its source code as well.

This implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing).
  1. The source code of this side car is retired/archived. So, it is not in active development. https://github.com/kubernetes-retired/contrib/tree/master/election

So, what is the proper method to elect a leader with Kubernetes?

-- SRF
kubernetes

0 Answers