Kubernetes' High Availability Leader Lease

9/28/2018

I have a question regarding Kubernetes' leader/follower lease management for the kube-controller-manager and the kube-scheduler: As far as I understand Kubernetes tracks the current leader as Endpoints in the kube-system namespace.

You can get the leader via

$ kubectl get endpoints -n kube-system

NAME                      ENDPOINTS                   AGE
kube-controller-manager   <none>                      20m
kube-scheduler            <none>                      20m

then e.g.

$ kubectl describe endpoints kube-scheduler -n kube-system

Name:         kube-scheduler
Namespace:    kube-system
Annotations:  control-plane.alpha.kubernetes.io/leader={"holderIdentity":"controller-0", ...}

The current leader is the holderIdentity of the control-plane.alpha.kubernetes.io/leader annotation.

My question:

Lease management like acquiring leases, renewing leases, time to live, etc. is implemented in leaderelection.go on top of Kubernetes Endpoints. Is there a specific reason lease management is not implemented directly on Etcd with "out-of-the-box" Etcd primitives like Etcd's compare and swap operation and time to live on objects?

Edit(s)

  • add Etcd compare and swap
  • add Etcd time to live
-- dtornow
etcd
high-availability
kubernetes

2 Answers

9/29/2018

A few reasons:

  1. Etcd might be running externally to Kubernetes' network, which means network latency
  2. Etcd could be busy/loaded and therefore slow
  3. The etcd cluster is very likely to have fewer nodes than the Kubernetes master, making it less reliable
-- samhain1138
Source: StackOverflow

4/24/2020

For security reasons, only the API server should have access to etcd. Keep in mind that if etcd was used for leader leases by convention, custom controllers and operators using leader election would also need access to etcd which would be inadvisable given how critical the data stored in etcd is.

Ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#securing-etcd-clusters

-- lander2k2
Source: StackOverflow