I would like to present a possible scenario in Kubernetes. There is a service S, which is sustained by 3 PODS X, Y and Z (thanks to the replication controller). When a client X contacts S it will be redirected (by iptables) to one of the PODS X, Y or Z. At any time, if some POD (X, Y or Z) fails, that POD will be restarted/recovered by Kubernetes.
The hypothesis: client X accessed the service S and was redirected to POD X. During this access, POD X fails. The client X will be notified of this failure (like: connection reset by peer), and the client X will have to contact again the service S, to be redirected to some POD that will answer the request.
Is the hypothesis true?
Therefore, here is question: in Kubernetes, is there some way to masquerade this failure from the client?
Thanks in advance, Hylson
Two thoughts:
You can set your pod health checks to be rather quick to minimize the time the service is down before the load balancer gets updated.
Also, I don't know if this exists, but you might be able to direct the load balancer (eg: AWS ELB) to retry upon failure condition on behalf of the client before returning error message to the user. But this is just a pie in the sky idea.