I am in the process of implementing an HA solution for Kubernetes Master nodes in a CentOS7 env.

My env looks like :

K8S_Master1 : 172.16.16.5
K8S_Master2 : 172.16.16.51
HAProxy     : 172.16.16.100
K8S_Minion1 : 172.16.16.50


etcd Version: 3.1.7
Kubernetes v1.5.2
CentOS Linux release 7.3.1611 (Core)

My etcd cluster is setup properly and is in working state.

[root@master1 ~]# etcdctl cluster-health
member 282a4a2998aa4eb0 is healthy: got healthy result from http://172.16.16.51:2379
member dd3979c28abe306f is healthy: got healthy result from http://172.16.16.5:2379
member df7b762ad1c40191 is healthy: got healthy result from http://172.16.16.50:2379

My K8S config for Master1 is :

[root@master1 ~]# cat /etc/kubernetes/apiserver 
KUBE_API_ADDRESS="--address=0.0.0.0"
KUBE_ETCD_SERVERS="--etcd_servers=http://127.0.0.1:4001"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.100.0.0/16"
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

[root@master1 ~]# cat /etc/kubernetes/config 
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow_privileged=false"
KUBE_MASTER="--master=http://127.0.0.1:8080"

[root@master1 ~]# cat /etc/kubernetes/controller-manager 
KUBE_CONTROLLER_MANAGER_ARGS="--leader-elect"

[root@master1 ~]# cat /etc/kubernetes/scheduler 
KUBE_SCHEDULER_ARGS="--leader-elect"

As for Master2 , I have configured it to be :

[root@master2 kubernetes]# cat apiserver 
KUBE_API_ADDRESS="--address=0.0.0.0"
KUBE_ETCD_SERVERS="--etcd_servers=http://127.0.0.1:4001"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.100.0.0/16"
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

[root@master2 kubernetes]# cat config 
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow_privileged=false"
KUBE_MASTER="--master=http://127.0.0.1:8080"

[root@master2 kubernetes]# cat scheduler 
KUBE_SCHEDULER_ARGS=""

[root@master2 kubernetes]# cat controller-manager 
KUBE_CONTROLLER_MANAGER_ARGS=""

Note that --leader-elect is only configured on Master1 as I want Master1 to be the leader.

My HA Proxy config is simple :

frontend K8S-Master
    bind 172.16.16.100:8080
    default_backend K8S-Master-Nodes

backend K8S-Master-Nodes
    mode        http
    balance     roundrobin
    server      master1 172.16.16.5:8080 check
    server      master2 172.16.16.51:8080 check

Now I have directed my minion to connect to the Load Balancer IP rather than directly to the Master IP.

Config on Minion is :

[root@minion kubernetes]# cat /etc/kubernetes/config 
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow_privileged=false"
KUBE_MASTER="--master=http://172.16.16.100:8080"

On both Master nodes, I see the minion/node status as Ready

[root@master1 ~]# kubectl get nodes
NAME           STATUS    AGE
172.16.16.50   Ready     2h

[root@master2 ~]# kubectl get nodes
NAME           STATUS    AGE
172.16.16.50   Ready     2h

I setup an example nginx pod using :

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

I created the Replication Controller on Master1 using :

[root@master1 ~]# kubectl create -f nginx.yaml

And on both Master nodes, I was able to see the pods created.

[root@master1 ~]# kubectl get po
NAME          READY     STATUS    RESTARTS   AGE
nginx-jwpxd   1/1       Running   0          29m
nginx-q613j   1/1       Running   0          29m

[root@master2 ~]# kubectl get po
NAME          READY     STATUS    RESTARTS   AGE
nginx-jwpxd   1/1       Running   0          29m
nginx-q613j   1/1       Running   0          29m

Now logically thinking, if I were to take down Master1 node and delete the pods on Master2 , Master2 should recreate the pods. So this is what I do.

On Master1 :

[root@master1 ~]# systemctl stop kube-scheduler ; systemctl stop kube-apiserver ; systemctl stop kube-controller-manager

On Master2 :

[root@slave1 kubernetes]# kubectl delete po --all
pod "nginx-l7mvc" deleted
pod "nginx-r3m58" deleted

Now Master2 should create the pods since the Replication Controller is still up. But the new Pods are stuck in :

[root@master2 kubernetes]# kubectl get po
NAME          READY     STATUS        RESTARTS   AGE
nginx-l7mvc   1/1       Terminating   0          13m
nginx-qv6z9   0/1       Pending       0          13m
nginx-r3m58   1/1       Terminating   0          13m
nginx-rplcz   0/1       Pending       0          13m

Ive waited a long time but the pods are stuck in this state.

But when I restart the services on Master1 :

[root@master1 ~]# systemctl start kube-scheduler ; systemctl start kube-apiserver ; systemctl start kube-controller-manager

Then I see progress on Master1 :

NAME          READY     STATUS              RESTARTS   AGE
nginx-qv6z9   0/1       ContainerCreating   0          14m
nginx-rplcz   0/1       ContainerCreating   0          14m

[root@slave1 kubernetes]# kubectl get po
NAME          READY     STATUS    RESTARTS   AGE
nginx-qv6z9   1/1       Running   0          15m
nginx-rplcz   1/1       Running   0          15m

Why doesnt Master2 recreate the pods ? This is the confusion that I am trying to figure out. Ive come a long way to setup a fully function HA setup but seems like almost there only if I can figure out this puzzle.