I am having issues in my current Kubernetes minikube set up getting pods to connect to ClusterIP services. My current setup environment looks like this:

OS: Rocky Linux 8 Guest Hosted with VMware on a Windows 10 Machine
VMware has 'Virtualize Intel VT-x/EPT or AMD-V/RVI' enabled
Minikube (v1.24.0) is running with docker (Docker version 20.10.11, build dea9396) as its driver

To isolate the problem I have started using this simple golang hello world image. Simply put, if you wget url:8080 you will download an index.html.

After building the image locally I create a pod with:

kubectl run hello --image=hello --port=8080 --labels='app=hello'

The pod spins up fine, and I can exec into it. Inside the pod, if I run:

wget localhost:8080 or wget 172.17.0.3:8080

I get the expected output of:

converted 'http://172.17.0.3:8080' (ANSI_X3.4-1968) -> 'http://172.17.0.3:8080' (UTF-8)
--2022-01-09 20:15:44--  http://172.17.0.3:8080/
Connecting to 172.17.0.3:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13 [text/plain]
Saving to: 'index.html'

index.html                                   100%[==============================================================================================>]      13  --.-KB/s   in 0s     

2022-01-09 20:15:44 (3.11 MB/s) - 'index.html' saved [13/13]

Now, if I expose the pod with: kubectl expose pod hello --name=hello-service --port=8080 --target-port=8080 the service is started as hello-service and describing it outputs the following:

Name:              hello-service
Namespace:         default
Labels:            app=hello
Annotations:       <none>
Selector:          app=hello
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.101.73.45
IPs:               10.101.73.45
Port:              <unset>  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.3:8080
Session Affinity:  None
Events:            <none>

The ports are set and the Endpoint exists, so from everything I've read this should work. So I exec back into the pod and try to wget the service and I get:

root@hello:/go/src/app# wget hello-service:8080
converted 'http://hello-service:8080' (ANSI_X3.4-1968) -> 'http://hello-service:8080' (UTF-8)
--2022-01-09 20:36:06--  http://hello-service:8080/
Resolving hello-service (hello-service)... 10.101.73.45
Connecting to hello-service (hello-service)|10.101.73.45|:8080... failed: Connection timed out.

The same happens when I try wget 10.101.73.45:8080, which of course makes sense because hello-service resolved to the correct IP in the previous wget.

Now, I'm no expert at Kubernetes, obviously, but this next part is weird to me. If I instead expose the pod with a nodePort, everything works as you would expect. Using the following definition file:

apiVersion: v1
kind: Service
metadata:
  name: hello-service
spec:
  selector:
    app: hello
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
    nodePort: 31111
  type: NodePort

I can hit the pod from the nodePort. A simple wget 192.168.49.2:31111 and I get the expected output:

--2022-01-09 15:00:48--  http://192.168.49.2:31111/
Connecting to 192.168.49.2:31111... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13 [text/plain]
Saving to: ‘index.html’

index.html                                   100%[============================================================================================>]      13  --.-KB/s    in 0s      

2022-01-09 15:00:48 (3.05 MB/s) - ‘index.html’ saved [13/13]

Anyway, I'm at my amateur wits end here. It's been a few days of trying to find similar issues that we're not just "oh you did not label your container correctly" or "there is a typo in your port listings" with little luck. I think this situation is unique enough to warrant its post.