I have liveness and readiness checks failing with the following error visible in the pod description:

Readiness probe failed: dial tcp 10.123.1.23:5000: connect: connection refused

When I curl from a replica of the pod to https://10.123.1.23:5000, I get a similar error (Failed to connect to ...the IP.. port 5000: Connection refused). I read something suggesting that if I should be able to curl from another pod if the address/port is valid. I don't understand why that would be the case, as I don't fully understand the cluster network internals.

So I'm assuming that the address is actually invalid for the readiness probe. Why can't I connect with curl? The IP number matches the IP of my pod and I see this under Containers in the pod description:

Containers:
....
Port:           5000/TCP

This is my yaml showing the health probe setup:

apiVersion: apps/v1
kind: Deployment
metadata:
   name: myApp
spec:
   ... 
   template:
     metadata:
       labels:
         app: myApp
     spec:
        ...
        containers:
          - name: myApp
            ...
            ports:
              - containerPort: 5000          
            ...
            readinessProbe:
              tcpSocket:
                  port: 5000
              initialDelaySeconds: 300
              periodSeconds: 5
              successThreshold: 1
              failureThreshold: 3
            livenessProbe:
               tcpSocket:
                  port: 5000
               periodSeconds: 30 
               initialDelaySeconds: 300
               successThreshold: 1
               failureThreshold: 3

...

I also tried and http probe to call the index page but get a similar error Readiness probe failed: Get http://...myIP...:5000/index.html: dial tcp ...myIP...:5000: connect: connection refused

        readinessProbe:
          httpGet:
            path: /index.html
            port: 5000
          initialDelaySeconds: 500
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
           httpGet:
            path: /index.html
            port: 5000
           periodSeconds: 30 
           initialDelaySeconds: 500
           successThreshold: 1
           failureThreshold: 3