I am getting connection refused when communicating between microservices.

Microservice is exposed as Cluster IP on port 4004 via service discovery

DNS is resolved correctly:

/app # ping sms.obchod-uat.svc.cluster.local
PING sms.obchod-uat.svc.cluster.local (10.43.204.129): 56 data bytes

When I am trying curl to port 80 it is waiting to timeout, because microservice hasn't exposed this port - correct behaviour, but when I am trying to curl to exposed port it gaves me Connection refused.

/app # curl sms.obchod-uat.svc.cluster.local:4004
curl: (7) Failed to connect to sms.obchod-uat.svc.cluster.local port 4004 after 12 ms: Connection refused

So it means that service has correctly exposed port and I can request to it. But for some reason something rejects it.

I am using:

RKE **2.5.7**
kubernetes **1.20.11**
CoreDNS **1.8.0**

Service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/creatorId: user-5srkl
    field.cattle.io/ipAddresses: "null"
    field.cattle.io/targetDnsRecordIds: "null"
    field.cattle.io/targetWorkloadIds: '["deployment:obchod-uat:sms"]'
  creationTimestamp: "2022-01-22T13:30:30Z"
  labels:
    cattle.io/creator: norman
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:field.cattle.io/creatorId: {}
          f:field.cattle.io/ipAddresses: {}
          f:field.cattle.io/targetDnsRecordIds: {}
          f:field.cattle.io/targetWorkloadIds: {}
        f:labels:
          .: {}
          f:cattle.io/creator: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":4004,"protocol":"TCP"}:
            .: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:publishNotReadyAddresses: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: rancher
    operation: Update
    time: "2022-01-22T13:30:29Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:selector:
          .: {}
          f:workloadID_sms: {}
    manager: agent
    operation: Update
    time: "2022-01-22T13:30:30Z"
  name: sms
  namespace: obchod-uat
  resourceVersion: "57096530"
  uid: f6213174-dcad-4535-be2c-e933a03e2fa6
spec:
  clusterIP: 10.43.204.129
  clusterIPs:
  - 10.43.204.129
  ports:
  - port: 4004
    protocol: TCP
    targetPort: 4004
  publishNotReadyAddresses: true
  selector:
    workloadID_sms: "true"
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Any ideas?

SOLVED

If you are using nest microservices, don't forget to add host property in to options.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { MicroserviceOptions, Transport } from '@nestjs/microservices';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(
    AppModule,
    {
      transport: Transport.TCP,
      options: {
        host: '0.0.0.0',
        port: 4004,
      },
    },
  );

  await app.listen();
}
bootstrap();

It's taken 2 days of my life.