Exposing mongodb on kubernetes statefulsets to external world

6/23/2017

I have setup the mongodb replicaset on kubernetes with Statefulsets using this helm chart

I can access the mongo instance inside the cluster. But I'd like to open it for access to the external world. I tried two approaches to accomplish it

  1. Create an additional service of type 'NodePort' mapping to the mongo instances with the selector label.

  2. Expose all 3 mongodb pods externally.

    kubectl expose pods mongo-release-mongodb-replicaset-2 --type=NodePort

Here is my test script.

 from pymongo import MongoClient
 client = MongoClient('192.168.99.100',30738) #approach 1
 #client = MongoClient('mongodb://192.168.99.100:31455,192.168.99.100:31424,192.168.99.100:31569/?replicaSet=rs0') #approach 2
 db=client.test
 db.test.insert({"key1":1})
 values=db.test.find({'key1': 1})
 for value in values:
    print value

With the first approach, I get the following error which makes sense, since the external service is not consistently hitting the primary node in replicaset. Multiple attempts would eventually connect to the master and the write works.

File "/Library/Python/2.7/site-packages/pymongo/pool.py", line 552, in _raise_connection_failure
raise error
pymongo.errors.NotMasterError: not master

With the second approach, since we are accessing each pod directly by their IP:port, I expected it to work but it throws the following exception

pymongo.errors.ServerSelectionTimeoutError: mongo-release-mongodb-replicaset-0.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017: [Errno 8] nodename nor servname provided, or not known,mongo-release-mongodb-replicaset-2.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017: [Errno 8] nodename nor servname provided, or not known,mongo-release-mongodb-replicaset-1.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017: [Errno 8] nodename nor servname provided, or not known

From the error it appears that the DNS translation is causing issues? I looked at this question but didn't get much help out of it

I'm running out of ideas. Can anyone help with this issue? Any alternative solutions are appreciated as well.

Thanks

-- Srikanth
kubernetes
mongodb
mongodb-replica-set

1 Answer

6/24/2017

After spending some more time with the issue, I figured that the mongo endpoint in the above script was returning the replicaset's DNS' in the following format - 'mongo-release-mongodb-replicaset-0.mongo-release-mongodb-replicaset.default.svc.cluster.local:27017'. These addresses can only be resolved within the cluster namespace. Verified the same by running the following script in another pod

from pymongo import MongoClient
client = MongoClient('mongodb://mongo-release-mongodb-replicaset-0.mongo-release-mongodb-replicaset:27017,mongo-release-mongodb-replicaset-1.mongo-release-mongodb-replicaset:27017,mongo-release-mongodb-replicaset-2.mongo-release-mongodb-replicaset:27017/?replicaSet=rs0')
db=client.test
db.test.insert({"key1":1})
values=db.test.find({'key1': 1})
for value in values:
   print value
-- Srikanth
Source: StackOverflow