I have a deployment with 2 replicas of Nginx. It has only the liveness probe to monitor the health of the service. Due to high traffic, my liveness probe fails, the Nginx container is restarted, but the pod status running state and Pod condition is Ready. Due to that POD, IP is not removed the service endpoint and request were send to restarting pod which results in some failures.
As per container probes:
livenessProbe: Indicates whether the Container is running. If the liveness probe fails, the kubelet kills the Container, and the Container is subjected to its restart policy [...]
readinessProbe: Indicates whether the Container is ready to service requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod [...]
You need to add a readinessProbe to grant that the endpoints of unhealthy containers will be removed.
Readiness probes are configured similarly to liveness probes. The only difference is that you use the
readinessProbefield instead of thelivenessProbefield.