This morning something has happened with ingress traffic for kubernetes cluster. All websites sites/API:s respond with 502 server error problem.
No updates/changes has been made to cluster or pods. All pods are green and working and all hosts are healty. Everything has been working for more then 2 weeks until this morning.
Do anyone have knowledge with this kind of issue?
Update:
Found 'failed_to_pick_backend' in logs. Strange that could happen after 2-3 weeks..
There could be a thousand of reasons for the failure of pods. Kubernetes has a functionality of self-healing which reconcile till current state is not equal to the desired state.
Kubernetes has accepted it that pods will be crashed, even nodes will be failed.
I assume, In your case, Ingress controller is not able to find a healthy pod to deliver traffic. Thus, It's better to check the events of the Pods.
kubectl get events
If you want to check out the logs of the previously terminated container from pod POD_NAME. You can use the following command with -p flag
kubectl logs POD_NAME -c CONTAINER_NAME -p