when and where a service ip created for kubernetes static pods

8/29/2018

I'm using kubernetes 1.6.7 cluster, 3 nodes as master (node1, node2 and node3).

The apiserver cluster ip as bellow:

root@node1:/etc/kubernetes/manifests# kubectl get svc 
NAME                     CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes               10.233.0.1      <none>        443/TCP        14d

root@node1:/etc/kubernetes/manifests# kubectl get endpoints
kubernetes               172.16.10.11:6443,172.16.10.12:6443,172.16.10.13:6443            14d

The problem is when node2 was down, tiller pod can't access 10.233.0.1:443, logs as bellow:

[storage] 2018/08/28 06:46:34 listing all releases with filter
[storage/driver] 2018/08/28 06:46:34 list: failed to list: Get https://10.233.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 10.233.0.1:443: connect: no route to host

and the service kubernetes still has 3 endpoints, but i think the 172.16.10.12 should be delete from the endpoint list. iptables as bellow:

-A KUBE-SVC-LRLUS54FOYJDJ5GT -m comment --comment "default/wangxj35-nginx:nginx" -j KUBE-SEP-GGIOR777CPKVSIB7
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-ZX77NMWHNJWLXYWF

-A KUBE-SEP-CS4Z4M5BFAAWBFXQ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.11:6443
-A KUBE-SEP-MQZA2ZXHX6GRRY22 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.12:6443
-A KUBE-SEP-ZX77NMWHNJWLXYWF -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.13:6443

Could anyone help me where the static pods service created by kubernetes source code?

Thanks.

-- toper
kubernetes

1 Answer

8/29/2018

Controller.go creates the kubernetes service:

 // UpdateKubernetesService attempts to update the default Kube service. 
 func (c *Controller) UpdateKubernetesService(reconcile bool)> error { 
       // Update service & endpoint records.

Also, to work around this problem, kubernetes1.9.0 has added --endpoint-reconciler-type=lease to allow apiserver to track the alive apiserver, see issue51698, while for release below 1.9.0, you can unset the --apiserver-count flag to let alive apiserver to preempt the only one endpoint position.

-- Kun Li
Source: StackOverflow