Can't connect to database using ExternalName on GKE in an ExpressJS Application

7/17/2020

I want to connect multiple microservices hosted on Google Kubernetes Engine to a MongoDB Atlas database.

All Deployments and Services inside the cluster are on the gitlab-managed-apps namespace.

I have created the following ExternalName Service:

mongo-external-name.yaml

kind: Service
apiVersion: v1
metadata:
  name: mongo
  namespace: gitlab-managed-apps
spec:
  type: ExternalName
  externalName: db_name.mrud5.mongodb.net

To connect to MongoDB Atlas I tried to use the ExternalName since using db_name.mrud5.mongodb.net would try to resolve to an internal service:

//connect.js
mongoose.connect(`mongodb+srv://${ global.DB_USER }:${ global.DB_PASSWORD }@mongo.gitlab-managed-apps.svc.cluster.local/${ global.DB_NAME }?retryWrites=true&w=majority`, {
		useNewUrlParser: true,
		useCreateIndex: true,
		useUnifiedTopology: true,
		useFindAndModify: false
	});

As per the deployments, I have them inside a couple of Helm Charts. I thought it had something to do with the dnsPolicy: ClusterFirst and I attempted to set it to dnsPolicy: Default, but that made no difference.

Upon executing bash on one of the pods, I have tried nslookup as suggested by many other issues, this is the result:

bash-5.0# nslookup mongo.gitlab-managed-apps.svc.cluster.local
Server:         10.1.0.10
Address:        10.1.0.10:53

    mongo.gitlab-managed-apps.svc.cluster.local     canonical name = db_name.mrud5.mongodb.net

    mongo.gitlab-managed-apps.svc.cluster.local     canonical name = db_name.mrud5.mongodb.net

I figured that if the pod can see the CNAME records there must be something else that I am not doing right.

I was unable to ping mongo.gitlab-managed-apps.svc.cluster.local

bash-5.0# ping mongo.gitlab-managed-apps.svc.cluster.local
ping: bad address 'mongo.gitlab-managed-apps.svc.cluster.local'

My cluster's master version is 1.15.11-gke.15. I did not create the cluster with this version, upon creation, Gitlab automatically chose 1.14.10. After reading multiple issues on GitHub, I understood that there were some issues with kube-dns on version 1.14.9. I, therefore, ran the command below to try to fix this.

kubectl set image deployment/kube-dns -n kube-system \
  kubedns=gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10 \
  dnsmasq=gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10 \
  sidecar=gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10

As I thought, this was not the solution either. Desperate, I upgraded the cluster to 1.15.11-gke.15 and kube-dns with the above command. That was not the solution either.

I don't think it causes this problem but I have posted below my Ingress controller:

ingress-service.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "7200"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "7200"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "7200"
  name: ingress-service
  namespace: gitlab-managed-apps
spec:
  tls:
    - hosts:
        - my.host
      secretName: letsencrypt-prod
  rules:
    - host: my.host
      http:
        paths:
          - backend:
              serviceName: my-cluster-ip-service
              servicePort: 3000
            path: /?(.*)

Any suggestion is very much appreciated!

-- Valentin Constanda
google-kubernetes-engine
kube-dns
kubernetes
kubernetes-ingress
nginx-ingress

2 Answers

7/19/2020

I have not managed to get the database to work using an Atlas Cluster hosted using AWS, but I was able to connect to an Atlas Cluster hosted using GCP.

So as a temporary fix if anybody comes by this issue, just create an Atlas Cluster using GCP.

I will leave the question open as the issue was avoided, not solved.

-- Valentin Constanda
Source: StackOverflow

7/18/2020

Well, the ExternalName configuration is correct โœ…. Your cluster pods are giving you the right canonical name. However, db_name.mrud5.mongodb.net is not even pingable nor reachable. It's basically a TXT DNS record. I say this because I tried it myself with my DB๐Ÿ‘‡:

dig txt mycluster.vvcme.mongodb.net

; <<>> DiG 9.10.6 <<>> txt cluster0.vvcme.mongodb.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18012
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;cluster0.vvcme.mongodb.net.	IN	TXT

;; ANSWER SECTION:
mycluster.vvcme.mongodb.net. 60	IN	TXT	"authSource=admin&replicaSet=atlas-xxxxxx-shard-0"

;; Query time: 127 msec
;; SERVER: 10.240.246.53#53(10.240.246.53)
;; WHEN: Fri Jul 17 19:26:52 PDT 2020
;; MSG SIZE  rcvd: 105

This is used by the client to get all the shards but in essence, you can find the shards directly๐Ÿ‘‡:

dig srv _mongodb._tcp.mycluster.vvcme.mongodb.net

; <<>> DiG 9.10.6 <<>> srv _mongodb._tcp.mycluster.vvcme.mongodb.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5089
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;_mongodb._tcp.mycluster.vvcme.mongodb.net. IN SRV

;; ANSWER SECTION:
_mongodb._tcp.cluster0.vvcme.mongodb.net. 60 IN	SRV 0 0 27017 mycluster-shard-00-00.vvcme.mongodb.net. ๐Ÿ‘ˆ
_mongodb._tcp.cluster0.vvcme.mongodb.net. 60 IN	SRV 0 0 27017 mycluster-shard-00-01.vvcme.mongodb.net. ๐Ÿ‘ˆ
_mongodb._tcp.cluster0.vvcme.mongodb.net. 60 IN	SRV 0 0 27017 mycluster-shard-00-02.vvcme.mongodb.net. ๐Ÿ‘ˆ

...

Those shards are actually pingable and DNS resolvable๐Ÿ‘‡:

dig cluster0-shard-00-00.vvcme.mongodb.net

; <<>> DiG 9.10.6 <<>> cluster0-shard-00-00.vvcme.mongodb.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41225
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;cluster0-shard-00-00.vvcme.mongodb.net.	IN A

;; ANSWER SECTION:
mycluster-shard-00-00.vvcme.mongodb.net.	60 IN CNAME mtm-aws-use1-22-m0-14-shard-00-00.5yapb.mongodb.net.
mtm-aws-use1-22-m0-14-shard-00-00.5yapb.mongodb.net. 60	IN CNAME ec2-xx-xx-xx-xx.compute-1.amazonaws.com.
ec2-xx-xx-xx-xx.compute-1.amazonaws.com. 86400 IN A xx.xxx.xx.xx ๐Ÿ‘ˆ

...

All this stuff is available on the open internet and it's documented here. So you don't even need your ExternalName Kubernetes service. Unless for some reason you are running Atlas in your private VPC in which case you would have to configure/ping the appropriate records. โœŒ๏ธ

-- Rico
Source: StackOverflow