We have helm based k8s application where one of the pod (named "auth") Has code that call itself to do CRUD operation for resetting password.
Since it's calling itself, This operation is using endpoint URL http://localhost:8081/API/authorizations
Issue: When deploying same application at on-premises k8s cluster auth pod is not able to connect itself.
Error: {"api":"external","component":"auth","delay":900000000,"error":"Post http://localhost:8081/api/authorizations: dial tcp 10.250.44.31:8081: connect: connection refused","insecureSkipVerify":false,"message":"Request failed, retrying","method":"POST","requestID":"fb9475e8-bcf9-4924-8339-d3d21922f357","severity":"debug","timestamp":"2020-02-10T11:54:29Z","url":"http://localhost:8081/api/authorizations"}
Note: 10.250.44.31:8081 is cluster endpoint IP address
Expected behavior: When running this application at AWS eks or azure aks, it's working fine with desired operation.
Question:
1) is it expected behavior for pod to connect to k8s master server ha proxy IP when using localhost endpoint inside pod?
2) for AWS eks and azure aks it's referring to pod URL only, so does it mean that our cluster network has to be troubleshooted for solving this issue?
Thanks with added ∆ points in advance..
what's the contents of /etc/hosts on the image being run? I'm guessing that it's misconfigured or missing, so the container process instead of getting proper IP of localhost from there, asks nameserver for it. probably your api server is also your cluster DNS and when asked about localhost, it responds with its own IP.
as a quick workaround you can change your program to use 127.0.0.1 directly instead of localhost.
1) is it expected behavior for pod to connect to k8s master server ha proxy IP when using localhost endpoint inside pod?
NO, It is not expected to connect to Master API Server for localhost communication
2) for AWS eks and azure aks it's referring to pod URL only, so does it mean that our cluster network has to be troubleshooted for solving this issue?
Exec into the pod
kubectl exec -it "pod-name" -n "namespace" bash
Once inside the pod, check if some process is listening on port 8081
ps -ef | grep "8081"
If you find the desired process running and listening on the port, curl the endpoint and check whether you are getting the response back or not