K
Q

Kubernetes: connection refused on the endpoint of a working pod

6/25/2019

I'm trying to debug why a service for a perfectly working deployment is not answering (connection refused).

I've double and tripled checked that the port and targetPort match (4180 for the container and 80 for the service)

when I list the my endpoints I get the following:

$ kubectl get endpoints
NAME           ENDPOINTS           AGE
kubernetes     10.40.63.79:443     82d
oauth2-proxy   10.40.34.212:4180   33s // <--this one

and from a pod running in the same namespace:

# curl 10.40.34.212:4180
curl: (7) Failed to connect to 10.40.34.212 port 4180: Connection refused

(By the way, same happens if I try to curl the service)

yet, if I port forward directly to the pod, I get a response:

$ kubectl port-forward oauth2-proxy-559dd9ddf4-8z72c 4180:4180 &
$ curl -v localhost:4180
* Rebuilt URL to: localhost:4180/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4180 (#0)
> GET / HTTP/1.1
> Host: localhost:4180
> User-Agent: curl/7.58.0
> Accept: */*
> 
Handling connection for 4180
< HTTP/1.1 403 Forbidden
< Date: Tue, 25 Jun 2019 07:53:19 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< 

<!DOCTYPE html>
// more of the expected response
* Connection #0 to host localhost left intact

I also checked that I get the pods when I use the selector from the service (I copy pasted it from what I see in kubectl describe svc oauth2-proxy):

$ kubectl describe svc oauth2-proxy | grep Selector
Selector:          app.kubernetes.io/name=oauth2-proxy,app.kubernetes.io/part-of=oauth2-proxy

$ kubectl get pods --selector=app.kubernetes.io/name=oauth2-proxy,app.kubernetes.io/part-of=oauth2-proxy
NAME                            READY   STATUS    RESTARTS   AGE
oauth2-proxy-559dd9ddf4-8z72c   1/1     Running   0          74m

I don't get why the endpoint is refusing the connection while using port forwarding gets a valid response. Anything else I should check?

-- Tom Klino
kubernetes

Similar Questions

Unable to assign public ip address to AKS: pending forever
Kubernetes Persistent Volume on GKE not mounting
hostname invalid when executing "kubeadm alpha certs xxx "
Flask application scaling on Kubernetes and Gunicorn

1 Answer

6/25/2019

Alright, turns out that this specific service was listening on localhost only by default:

$ netstat -tunap | grep LISTEN
tcp        0      0 127.0.0.1:4180          0.0.0.0:*               LISTEN      1/oauth2_proxy

I had to add an argument (-http-address=0.0.0.0:4180) to tell it to listen on 0.0.0.0

-- Tom Klino
Source: StackOverflow