in my kubernetes cluster, http liveness probe always failed with this message
Liveness probe failed: Get http://10.233.90.72:8080/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
so, coredns
and kubernetes-dashboard
(any other using http liveness probe) pods being infinitely restart.
while pod running (between events start and restart), i check the endpoints for the pod with executing command curl http://10.233.90.72:8080/health
on the busyboxplus
pod. this command are working normally, i can see OK
return. but liveness probe still failed. pod is restarting...
in this situation, i want to debug liveness probe, but i don't have any idea who/where actually work liveness probe in kubernetes? is this pod? or node?
how can i debug liveness probe? does anyone have same issue..?
please advice for me.
kubectl version:
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
version info:
OS: Ubuntu 18.04
Kubernetes: 1.13.3
Docker: 18.09.2
thanks in advance
Did you check the DNS already ? using the busybox:1.28 try to execute dns lookup to the pod and see what you get.
nslookup pod-ip-in-dash.pod.cluster.local
Another thing you can do which you might already did, check in kube-system if coredns pod is running.
Let me know how went,