I am having issues in my current Kubernetes minikube set up getting pods to connect to ClusterIP services. My current setup environment looks like this:
OS: Rocky Linux 8 Guest Hosted with VMware on a Windows 10 Machine VMware has 'Virtualize Intel VT-x/EPT or AMD-V/RVI' enabled Minikube (v1.24.0) is running with docker (Docker version 20.10.11, build dea9396) as its driver
To isolate the problem I have started using this simple golang hello world image. Simply put, if you
wget url:8080 you will download an index.html.
After building the image locally I create a pod with:
kubectl run hello --image=hello --port=8080 --labels='app=hello'
The pod spins up fine, and I can exec into it. Inside the pod, if I run:
wget localhost:8080 or
I get the expected output of:
converted 'http://172.17.0.3:8080' (ANSI_X3.4-1968) -> 'http://172.17.0.3:8080' (UTF-8) --2022-01-09 20:15:44-- http://172.17.0.3:8080/ Connecting to 172.17.0.3:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 13 [text/plain] Saving to: 'index.html' index.html 100%[==============================================================================================>] 13 --.-KB/s in 0s 2022-01-09 20:15:44 (3.11 MB/s) - 'index.html' saved [13/13]
Now, if I expose the pod with:
kubectl expose pod hello --name=hello-service --port=8080 --target-port=8080 the service is started as
hello-service and describing it outputs the following:
Name: hello-service Namespace: default Labels: app=hello Annotations: <none> Selector: app=hello Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.101.73.45 IPs: 10.101.73.45 Port: <unset> 8080/TCP TargetPort: 8080/TCP Endpoints: 172.17.0.3:8080 Session Affinity: None Events: <none>
The ports are set and the Endpoint exists, so from everything I've read this should work. So I exec back into the pod and try to wget the service and I get:
root@hello:/go/src/app# wget hello-service:8080 converted 'http://hello-service:8080' (ANSI_X3.4-1968) -> 'http://hello-service:8080' (UTF-8) --2022-01-09 20:36:06-- http://hello-service:8080/ Resolving hello-service (hello-service)... 10.101.73.45 Connecting to hello-service (hello-service)|10.101.73.45|:8080... failed: Connection timed out.
The same happens when I try
wget 10.101.73.45:8080, which of course makes sense because hello-service resolved to the correct IP in the previous wget.
Now, I'm no expert at Kubernetes, obviously, but this next part is weird to me. If I instead expose the pod with a nodePort, everything works as you would expect. Using the following definition file:
apiVersion: v1 kind: Service metadata: name: hello-service spec: selector: app: hello ports: - protocol: TCP port: 8080 targetPort: 8080 nodePort: 31111 type: NodePort
I can hit the pod from the nodePort. A simple
wget 192.168.49.2:31111 and I get the expected output:
--2022-01-09 15:00:48-- http://192.168.49.2:31111/ Connecting to 192.168.49.2:31111... connected. HTTP request sent, awaiting response... 200 OK Length: 13 [text/plain] Saving to: ‘index.html’ index.html 100%[============================================================================================>] 13 --.-KB/s in 0s 2022-01-09 15:00:48 (3.05 MB/s) - ‘index.html’ saved [13/13]
Anyway, I'm at my amateur wits end here. It's been a few days of trying to find similar issues that we're not just "oh you did not label your container correctly" or "there is a typo in your port listings" with little luck. I think this situation is unique enough to warrant its post.