I have 2 pods with simple echo app that are registered to the service that expose ELB in AWS.
I'm starting curl in a loop from my laptop to hit echo app and then I do kubectl delete pod echo-xyz
Results are wired as for next couple of seconds traffic is still being routed to deleted pod. Pod has been removed from endpoint list correctly though. The same test but issuing curl from inside a cluster to bypass ELB is working as expected - the moment i delete pod, traffic is being routed to single pod remaining until new one is created.
In the attached picture you can see on the left the test from inside a cluster and on the right from external via ELB.
K8s 1.16.8 installed via kops on AWS.
Any hint would be much appreciated.
The deletion of pods are not immediate. There are lists of events performed(wait for the grace period, preStop hooks, etc) by kubelet to bring down the pod once disruptions happen. Docs here: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
This process of graceful termination is by design that Kubernetes does not send KILL command, instead works with TERM(termination) signal.
Refer docs: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
Also, --force delete is not suggested as it updates the API server without waiting for the resource to be deleted.
