I have set up a AWS kops cluster for Kubernetes, i have multiple microservices where in each application need to interact with one another.

Scenario: My ta2carbon app tries to invoke a function in ta1carbon app through service(dns) name.

Result: It is failing with timeout error by trying to hit to port 80 (but configured port -3000)

my nodejs app console log, apiUrl: http://ta1carbon/api/app1/app1Func2

{ Error: connect ETIMEDOUT 100.66.7.165:80
    at Object._errnoException (util.js:992:11)
    at _exceptionWithHostPort (util.js:1014:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1186:14)
  code: 'ETIMEDOUT',
  errno: 'ETIMEDOUT',
  syscall: 'connect',
  address: '100.66.7.165',
  port: 80 }

same error logs for curl, when i tried to curl my ta1carbon app inside ta2carbon pod.

root@ta2carbon-5fdcfb97cc-8j4nl:/home/appHome# curl -i http://ta1carbon/api/app1/app1Func2
curl: (7) Failed to connect to ta1carbon port 80: Connection timed out

but my defined port in service.yaml is 3000 not 80! below are the yml configurations for service for both microservices.

ta1carbon service yaml

apiVersion: v1
kind: Service
metadata:
  name: ta1carbon
  labels:
    app: ta1carbon
spec:
  ports:
  - port: 3000
    targetPort: 3000
  type: ClusterIP
  selector:
    app: ta1carbon

ta2carbon service yaml

apiVersion: v1
kind: Service
metadata:
  name: ta2carbon
  labels:
    app: ta2carbon
spec:
  ports:
  - port: 3001
    targetPort: 3001
  type: ClusterIP
  selector:
    app: ta2carbon

And below is the describe service details for both ta1carbon and ta2 carbon.

kubectl describe service ta1carbon
Name:              ta1carbon
Namespace:         default
Labels:            app=ta1carbon
Annotations:       <none>
Selector:          app=ta1carbon
Type:              ClusterIP
IP:                100.66.7.165
Port:              <unset>  3000/TCP
TargetPort:        3000/TCP
Endpoints:         100.96.1.13:3000
Session Affinity:  None
Events:            <none>

kubectl describe service ta2carbon
Name:              ta2carbon
Namespace:         default
Labels:            app=ta2carbon
Annotations:       <none>
Selector:          app=ta2carbon
Type:              ClusterIP
IP:                100.67.129.126
Port:              <unset>  3001/TCP
TargetPort:        3001/TCP
Endpoints:         100.96.1.12:3001
Session Affinity:  None
Events:            <none>

So based on what i observe, for the url http://ta1carbon/api/app1/app1Func2 service dns ta1carbon is being resolved in to 100.67.24.69:80 resulting in timeout.

but however if i curl in to 100.67.24.69:3000 from inside ta2carbon pod i get a success response

Also if i change my service yaml - port: 80 and deploy and test again i get success response

i am finding this behaviour in kubernetes quite weird, not sure weather i am making mistake or with environment.

My query is -

why is it resolving service ta1carbon in to 100.67.24.69:80 and timing out, when the port should have been 3000!

Any input on this will be much appreciated. please let me know what is missing in this.