I want to connect multiple microservices hosted on Google Kubernetes Engine to a MongoDB Atlas database.
All Deployments and Services inside the cluster are on the gitlab-managed-apps
namespace.
I have created the following ExternalName Service:
mongo-external-name.yaml
kind: Service
apiVersion: v1
metadata:
name: mongo
namespace: gitlab-managed-apps
spec:
type: ExternalName
externalName: db_name.mrud5.mongodb.net
To connect to MongoDB Atlas I tried to use the ExternalName since using db_name.mrud5.mongodb.net would try to resolve to an internal service:
//connect.js
mongoose.connect(`mongodb+srv://${ global.DB_USER }:${ global.DB_PASSWORD }@mongo.gitlab-managed-apps.svc.cluster.local/${ global.DB_NAME }?retryWrites=true&w=majority`, {
useNewUrlParser: true,
useCreateIndex: true,
useUnifiedTopology: true,
useFindAndModify: false
});
As per the deployments, I have them inside a couple of Helm Charts. I thought it had something to do with the dnsPolicy: ClusterFirst
and I attempted to set it to dnsPolicy: Default
, but that made no difference.
Upon executing bash on one of the pods, I have tried nslookup
as suggested by many other issues, this is the result:
bash-5.0# nslookup mongo.gitlab-managed-apps.svc.cluster.local
Server: 10.1.0.10
Address: 10.1.0.10:53
mongo.gitlab-managed-apps.svc.cluster.local canonical name = db_name.mrud5.mongodb.net
mongo.gitlab-managed-apps.svc.cluster.local canonical name = db_name.mrud5.mongodb.net
I figured that if the pod can see the CNAME records there must be something else that I am not doing right.
I was unable to ping mongo.gitlab-managed-apps.svc.cluster.local
bash-5.0# ping mongo.gitlab-managed-apps.svc.cluster.local
ping: bad address 'mongo.gitlab-managed-apps.svc.cluster.local'
My cluster's master version is 1.15.11-gke.15
. I did not create the cluster with this version, upon creation, Gitlab automatically chose 1.14.10
. After reading multiple issues on GitHub, I understood that there were some issues with kube-dns
on version 1.14.9
. I, therefore, ran the command below to try to fix this.
kubectl set image deployment/kube-dns -n kube-system \
kubedns=gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10 \
dnsmasq=gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10 \
sidecar=gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10
As I thought, this was not the solution either. Desperate, I upgraded the cluster to 1.15.11-gke.15
and kube-dns with the above command. That was not the solution either.
I don't think it causes this problem but I have posted below my Ingress controller:
ingress-service.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/proxy-body-size: "100m"
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-connect-timeout: "7200"
nginx.ingress.kubernetes.io/proxy-read-timeout: "7200"
nginx.ingress.kubernetes.io/proxy-send-timeout: "7200"
name: ingress-service
namespace: gitlab-managed-apps
spec:
tls:
- hosts:
- my.host
secretName: letsencrypt-prod
rules:
- host: my.host
http:
paths:
- backend:
serviceName: my-cluster-ip-service
servicePort: 3000
path: /?(.*)
Any suggestion is very much appreciated!
I have not managed to get the database to work using an Atlas Cluster hosted using AWS, but I was able to connect to an Atlas Cluster hosted using GCP.
So as a temporary fix if anybody comes by this issue, just create an Atlas Cluster using GCP.
I will leave the question open as the issue was avoided, not solved.
Well, the ExternalName
configuration is correct โ
. Your cluster pods are giving you the right canonical name. However, db_name.mrud5.mongodb.net
is not even pingable nor reachable. It's basically a TXT DNS record. I say this because I tried it myself with my DB๐:
dig txt mycluster.vvcme.mongodb.net
; <<>> DiG 9.10.6 <<>> txt cluster0.vvcme.mongodb.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18012
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cluster0.vvcme.mongodb.net. IN TXT
;; ANSWER SECTION:
mycluster.vvcme.mongodb.net. 60 IN TXT "authSource=admin&replicaSet=atlas-xxxxxx-shard-0"
;; Query time: 127 msec
;; SERVER: 10.240.246.53#53(10.240.246.53)
;; WHEN: Fri Jul 17 19:26:52 PDT 2020
;; MSG SIZE rcvd: 105
This is used by the client to get all the shards but in essence, you can find the shards directly๐:
dig srv _mongodb._tcp.mycluster.vvcme.mongodb.net
; <<>> DiG 9.10.6 <<>> srv _mongodb._tcp.mycluster.vvcme.mongodb.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5089
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;_mongodb._tcp.mycluster.vvcme.mongodb.net. IN SRV
;; ANSWER SECTION:
_mongodb._tcp.cluster0.vvcme.mongodb.net. 60 IN SRV 0 0 27017 mycluster-shard-00-00.vvcme.mongodb.net. ๐
_mongodb._tcp.cluster0.vvcme.mongodb.net. 60 IN SRV 0 0 27017 mycluster-shard-00-01.vvcme.mongodb.net. ๐
_mongodb._tcp.cluster0.vvcme.mongodb.net. 60 IN SRV 0 0 27017 mycluster-shard-00-02.vvcme.mongodb.net. ๐
...
Those shards are actually pingable and DNS resolvable๐:
dig cluster0-shard-00-00.vvcme.mongodb.net
; <<>> DiG 9.10.6 <<>> cluster0-shard-00-00.vvcme.mongodb.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41225
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;cluster0-shard-00-00.vvcme.mongodb.net. IN A
;; ANSWER SECTION:
mycluster-shard-00-00.vvcme.mongodb.net. 60 IN CNAME mtm-aws-use1-22-m0-14-shard-00-00.5yapb.mongodb.net.
mtm-aws-use1-22-m0-14-shard-00-00.5yapb.mongodb.net. 60 IN CNAME ec2-xx-xx-xx-xx.compute-1.amazonaws.com.
ec2-xx-xx-xx-xx.compute-1.amazonaws.com. 86400 IN A xx.xxx.xx.xx ๐
...
All this stuff is available on the open internet and it's documented here. So you don't even need your ExternalName
Kubernetes service. Unless for some reason you are running Atlas in your private VPC in which case you would have to configure/ping the appropriate records. โ๏ธ