mongo connection error inside mongoS container

3/11/2019

I'm trying to orchestrate a MongoDB cluster inside Kubernetes, working with StatefulSets and Headless Services. It's been a while I'm trying to figure out what's wrong in this but can't find anything I could use.

While doing my research I found this repository which I cloned and adapted to the use case. You can find one of the manifests (the one for the mongoS instance) right here.

I then run the generate.sh script shipped within the repository, but with everything related to gcloud or persistentVolume commented, leaving only the statefulSet/Service spawning processes. Everything seems to be working fine, but whenever the script reaches the block

# Wait for the mongos to have started properly
echo "Waiting for the first mongos to come up (`date`)..."
echo " (IGNORE any reported not found & connection errors)"
echo -n "  "
until kubectl --v=0 exec mongos-router-0 -c mongos-container -- mongo --quiet --eval 'db.getMongo()'; do
    sleep 2
    echo -n "  "
done
echo "...first mongos is now running (`date`)"
echo

It just gives this error message indifenitely:

2019-03-11T14:02:03.531+0000 E QUERY    [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:343:13
@(connect):1:21
exception: connect failed
command terminated with exit code 1

Thus I tried to run mongo from inside the pod directly, but the same error happened. Then I just tried to run mongo using different ports just inside the container, running mongo --port 27018 (also tried 27019 and 27020), just to see which kind of errors it gives with non-exposed ports, and it seems that it gives the same error.

How could I run mongo inside a container with a running mongos on port 27017? It seems that I have to run mongod before running the mongo shell, even if mongos already started. It's really mind-boggling and I feel like I'm missing something. I only have this error inside the mongos pods, while the mongo-configdb and mongod-maindb seems to be running smoothly (and running the mongo shell inside them doesn't give any errors. Moreover, when doing rs.status() inside one of those it works fine).

You can find some mongos logs there.

TL;DR How could I run mongo inside a container with a running mongos on port 27017?

N.B.: I'm using the image mongo:3.6 from the official dockerhub repository.

-- Yoiro
kubernetes
mongo-shell
mongodb

1 Answer

3/11/2019

I don't really understand why, but changing the running port from 27017 to 27020 worked fine. If someone passes by and can explain how, I'd be glad to hear his/her explanation as netstat -tunelp (as well a ps aux) while grepping port 27017 on both of my k8s nodes does not give any output related to port 27017...

-- Yoiro
Source: StackOverflow