Kubernetes service not able to curl itself inside docker

1/28/2019

Background
I am testing the Kubernetes setting on Minikube. I have two simple services successfully setup and they are backed by simple docker image. Below is an example of my service configuration. I use NodePort to expose the service on port 80.

# service 1
kind: Service
apiVersion: v1
metadata:
  name: service1
spec:
  selector:
    app: service1
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service1-deployment
  labels:
    app: service1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service1
  template:
    metadata:
      labels:
        app: service1
    spec:
      containers:
      - name: service1
        image: service1
        imagePullPolicy: Never
        ports:
        - containerPort: 8080
---
# service 2
kind: Service
apiVersion: v1
metadata:
  name: service2
spec:
  selector:
    app: service2
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service2-deployment
  labels:
    app: service2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service2
  template:
    metadata:
      labels:
        app: service2
    spec:
      containers:
      - name: service2
        image: service2
        imagePullPolicy: Never
        ports:
        - containerPort: 8080

Issue
I use docker exec -it to go inside docker container. I can curl service1 from service2 container without any issue. However, if I try to curl service2 from service2 container, it gets a timeout connection error.

Results from curl -v service2

Rebuilt URL to: service2/
Trying 10.101.116.46...
TCP_NODELAY set
connect to 10.101.116.46 port 80 failed: Connection timed out
Failed to connect to service2 port 80: Connection timed out
Closing connection 0
curl: (7) Failed to connect to service2 port 80: Connection timed out

I guess the DNS records gets resolved correctly, because 10.101.116.46 is the correct IP attached to service2. Then what could be the issue cause this problem?

More Followup Tests
From my understanding, the Kubernetes service internally maps the port to container port, so in my case it maps service port 80 to pod port 8080. From service2 container, I am able to curl <service2 pod ip>:8080 successfully, but I am not able to curl <service2 ip>, which resolves connection time out error. And this happens exactly the same inside the service1 container that it can access pod but no service. I do not understand is there any internal setting that I miss?

-- Jiashen Cao
docker
kubernetes

1 Answer

1/28/2019

This could be any of these:

  • The pod servicing service2 has a service that is listening on 127.0.0.1 or not listening on 0.0.0.0 (Any IP address)
  • service2 has a redirect and your service only listen on port 80. You would have to enable the other port (possibly 443) and run curl with the -L option to follow the link.
  • The pod servicing service2 is not even listening on port 80.
-- Rico
Source: StackOverflow