Cannot access pod via pod IP from within the cluster, causing liveness / readiness probe failure

2/27/2019

Some quick background: creating an app in golang, running on minikube on MacOS 10.14.2

karlewr [0] $ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-18T11:37:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.4", GitCommit:"f49fa022dbe63faafd0da106ef7e05a29721d3f1", GitTreeState:"clean", BuildDate:"2018-12-14T06:59:37Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

The Issue: I cannot access my pod via it's pod IP from inside my cluster. This problem is only happening with this one pod which leads me to believe I have a misconfiguration somewhere.

My pods spec is as follows:

containers:
  - name: {{ .Chart.Name }}
    image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
    ports:
      - name: http
        containerPort: 8080
        protocol: TCP
    livenessProbe:
      httpGet:
        path: /ping
        port: 8080
      initialDelaySeconds: 60
    readinessProbe:
      httpGet:
        path: /ping
        port: 8080
      initialDelaySeconds: 60

What's weird is that I can access it by port-forwarding that pod on port 8080 and running curl localhost:8080/ping before the liveness and readiness probes run and after the pod has been initialized. This returns 200 OK.

Also during this time before CrashLoopBackoff, if I ssh into my minikube node and run curl http://172.17.0.21:8080/ping I get curl: (7) Failed to connect to 172.17.0.21 port 8080: Connection refused. The IP used is my pod's IP.

But then when I describe the pod after the initialDelaySeconds period, I see this:

  Warning  Unhealthy  44s (x3 over 1m)  kubelet, minikube  Readiness probe failed: Get http://172.17.0.21:8080/ping: dial tcp 172.17.0.21:8080: connect: connection refused
  Warning  Unhealthy  44s (x3 over 1m)  kubelet, minikube  Liveness probe failed: Get http://172.17.0.21:8080/ping: dial tcp 172.17.0.21:8080: connect: connection refused

Why would my connection be getting refused only from the pod's IP?

Edit I am not running any custom networking things, just minikube out of the box

-- rkarlewicz
docker
kubernetes
minikube

1 Answer

2/27/2019

Why would my connection be getting refused only from the pod's IP?

Because your program is apparently only listening on localhost (aka 127.0.0.1 aka lo0)

Without knowing more about your container we can't advise you further, but that's almost certainly the problem based on your description.

-- mdaniel
Source: StackOverflow