I'm running the leader-elector (v0.5) as a sidecar on three pods, on three different nodes.
Arguments: --election=XXX --http=0.0.0.0:4040
All works well until I kill the leader pod.
Now I get one of the pods into a state where the logs say it switched to the new leader:
kubectl logs -c elector upper-0
I0803 21:08:38.645849 7 leaderelection.go:296] lock is held by upper-1 and has not yet expired
So that indicates that upper-1 is now the leader.
But if I do query the HTTP server of upper-0, it returns the old leader:
kubectl exec -it upper-0 bash
root@nso-lsa-upper-0:/# curl http://localhost:4040
{"name":"upper-2"}
Do I need to do something for the leader-electors HTTP service to update?
Yes, bug. I've uploaded a fixed container here: https://hub.docker.com/r/fredrikjanssonse/leader-elector/tags/
It looks like a bug.
There is an open issue on GitHub: How is this possible? #2930