when delete a pod, kubernetes first deletes it in etcd via apiserver, then the controllers and kubelet do some stuff based on the changing of objects storaged in etcd, am i right? so here comes the question, after a pod has been deleted in etcd, the endpoint controller and kubelet both should react, but which one will complete first? if the pod has been actually killed by kubelet on node, and the endpoint has not, as a result, some visits to the service will be lost. is that right? thanks!
I think you are right. It's possible that a Pod is deleted before the endpoints are updated. Once a pod's deletionTimeStamp(https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/pods.go#L71) is set, kubelet will signal the container(s) to stop, and the endpoints controller will start updating the relevant endpoints. It's random which one will finish first.
[Update]
when delete a pod, kubernetes first deletes it in etcd via apiserver, then the controllers and kubelet do some stuff based on the changing of objects storaged in etcd, am i right
If a grace period (30s by default) is specified, the pod will be removed from the list of endpoints first, and deleted from the api server after the grace period expires.
if the pod has been actually killed by kubelet on node, and the endpoint has not, as a result, some visits to the service will be lost. is that right?
This section in the docs is particularly useful for this topic: https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/pods.md#termination-of-pods
Because the processes in the Pod being deleted are sent the TERM signal, so they have a chance to finish serving the ongoing requests, so if you delete the pod with a grace period (30s by default), and if the processes handles the TERM signal properly, the pending requests won't be lost.