While trying to setup Cassandra database in a local Kubernetes cluster on a Mac OS (via Minikube), I am getting connection issues. It seems like Node.js is not able to resolve DNS settings correctly, but resolving via command line DOES work.

The setup is as following (simplified): Cassandra Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  type: NodePort
  ports:
    - port: 9042
      targetPort: 9042
      protocol: TCP
      name: http
  selector:
    app: cassandra

In addition, there's a PersistentVolume and a StatefulSet.

The application itself is very basic

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: app1
  labels:
    app: app1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
      - name: app1
        image: xxxx.dkr.ecr.us-west-2.amazonaws.com/acme/app1
        imagePullPolicy: "Always"
        ports:
        - containerPort: 3003

And a service

apiVersion: v1
kind: Service
metadata:
  name: app1
  namespace: default
spec:
  selector:
    app: app1
  type: NodePort
  ports:
    - port: 3003
      targetPort: 3003
      protocol: TCP
      name: http

there also a simple ingress setup

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: dev.acme.com
    http:
      paths:
      - path: /app1
        backend:
          serviceName: app1
          servicePort: 3003

And added to /etc/hosts the minikube ip address

192.xxx.xx.xxx dev.acme.com

So far so good.

When trying to call dev.acme.com/app1 via Postman, the node.js app itself is being called correctly (can see in the logs), HOWEVER, the app can not connect to Cassandra and times out with the following error:

"All host(s) tried for query failed. First host tried, 92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."

The IP 92.242.140.2 seems to be just a public IP that is related to my ISP, I believe since the app is not able to resolve the service name.

I created a simple node.js script to test dns:

var dns = require('dns') dns.resolve6('cassandra', (err, res) => console.log('ERR:', err, 'RES:', res))

and the response is

ERR: { Error: queryAaaa ENOTFOUND cassandra at QueryReqWrap.onresolve [as oncomplete] (dns.js:197:19) errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'queryAaaa', hostname: 'cassandra' } RES: undefined

However, and this is where it gets confusing - when I ssh into the pod (app1), I am able to connect to cassandra service using:

cqlsh cassandra 9042 --cqlversion=3.4.4

So it seems as the pod is "aware" of the service name, but node.js runtime is not.

Any idea what could cause the node.js to not being able to resolve the service name/dns settings?

UPDATE

After re-installing the whole cluster, including re-installing docker, kubectl and minikube I am getting the same issue.

While running ping cassandra from app1 container via ssh, I am getting the following

PING cassandra.default.svc.cluster.local (10.96.239.137) 56(84) bytes of data. 64 bytes from cassandra.default.svc.cluster.local (10.96.239.137): icmp_seq=1 ttl=61 time=27.0 ms

2 packets transmitted, 2 received, 0% packet loss, time 1001ms

Which seems to be fine. However, when running from Node.js runtime I am still getting the same error -

"All host(s) tried for query failed. First host tried, 92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."

These are the services

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
app1        ClusterIP   None            <none>        3003/TCP         11m
cassandra    NodePort    10.96.239.137   <none>        9042:32564/TCP   38h
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          38h

And these are the pods (all namespaces)

NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
default       app1-85d889db5-m977z                       1/1     Running   0          2m1s
default       cassandra-0                                 1/1     Running   0          38h
kube-system   calico-etcd-ccvs8                           1/1     Running   0          38h
kube-system   calico-node-thzwx                           2/2     Running   0          38h
kube-system   calico-policy-controller-5bb4fc6cdc-cnhrt   1/1     Running   0          38h
kube-system   coredns-86c58d9df4-z8pr4                    1/1     Running   0          38h
kube-system   coredns-86c58d9df4-zcn6p                    1/1     Running   0          38h
kube-system   default-http-backend-5ff9d456ff-84zb5       1/1     Running   0          38h
kube-system   etcd-minikube                               1/1     Running   0          38h
kube-system   kube-addon-manager-minikube                 1/1     Running   0          38h
kube-system   kube-apiserver-minikube                     1/1     Running   0          38h
kube-system   kube-controller-manager-minikube            1/1     Running   0          38h
kube-system   kube-proxy-jj7c4                            1/1     Running   0          38h
kube-system   kube-scheduler-minikube                     1/1     Running   0          38h
kube-system   kubernetes-dashboard-ccc79bfc9-6jtgq        1/1     Running   4          38h
kube-system   nginx-ingress-controller-7c66d668b-rvxpc    1/1     Running   0          38h
kube-system   registry-creds-x5bhl                        1/1     Running   0          38h
kube-system   storage-provisioner                         1/1     Running   0          38h

UPDATE 2

The code to connect to Cassandra from Node.js:

const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['cassandra:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });

const query = 'SELECT * FROM user';
client.execute(query, [])
  .then(result => console.log('User with email %s', result.rows[0].email));

It DOES work when replacing cassandra:9042 with 10.96.239.137:9042 (10.69.239.137 is the ip address received from pinging cassandra via cli).