GCP Kubernetes MongoDb statefulset connection string MongooseError getaddrinfo ENOTFOUND

7/21/2020

I am following the following tutorial to configure MongoDB as a stateful set within my Kubernetes cluster on GCP.

https://codelabs.developers.google.com/codelabs/cloud-mongodb-statefulset/index.html?index=..%2F..index#0

I am able to access the database using "kubectl exec -ti mongo-0 mongo" as shown in the tutorial. However, my Node JS- Mongoose Application is unable to connect to it throwing the following error

MongoDB connection error: { MongooseError [MongooseServerSelectionError]: connect ECONNREFUSED 10.16.0.22:27017
    at new MongooseServerSelectionError (/usr/src/app/node_modules/mongoose/lib/error/serverSelection.js:24:11)
    at NativeConnection.Connection.openUri (/usr/src/app/node_modules/mongoose/lib/connection.js:823:32)
    at Mongoose.connect (/usr/src/app/node_modules/mongoose/lib/index.js:333:15)
    at Object.<anonymous> (/usr/src/app/app.js:6:10)
    at Module._compile (internal/modules/cjs/loader.js:816:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:827:10)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
    at Function.Module.runMain (internal/modules/cjs/loader.js:877:12)
    at internal/main/run_main_module.js:21:11
  message: 'connect ECONNREFUSED 10.16.0.22:27017',
  name: 'MongooseServerSelectionError',
  reason:
   TopologyDescription {
     type: 'ReplicaSetNoPrimary',
     setName: null,
     maxSetVersion: null,
     maxElectionId: null,
     servers:
      Map {
        'mongo-0.mongo:27017' => [ServerDescription],
        'mongo-1.mongo:27017' => [ServerDescription],
        'mongo-2.mongo:27017' => [ServerDescription] },
     stale: false,
     compatible: true,
     compatibilityError: null,
     logicalSessionTimeoutMinutes: null,
     heartbeatFrequencyMS: 10000,
     localThresholdMS: 15,
     commonWireVersion: null },
  [Symbol(mongoErrorContextSymbol)]: {} }
(node:29) UnhandledPromiseRejectionWarning: MongooseServerSelectionError: connect ECONNREFUSED 10.16.0.22:27017
    at new MongooseServerSelectionError (/usr/src/app/node_modules/mongoose/lib/error/serverSelection.js:24:11)
    at NativeConnection.Connection.openUri (/usr/src/app/node_modules/mongoose/lib/connection.js:823:32)
    at Mongoose.connect (/usr/src/app/node_modules/mongoose/lib/index.js:333:15)
    at Object.<anonymous> (/usr/src/app/app.js:6:10)
    at Module._compile (internal/modules/cjs/loader.js:816:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:827:10)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
    at Function.Module.runMain (internal/modules/cjs/loader.js:877:12)
    at internal/main/run_main_module.js:21:11
(node:29) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:29) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

app.js connection code is as follows

mongoose.connect(
  process.env.MONGO_CONNECTION_STRING,
  {
    useUnifiedTopology: true,
    useNewUrlParser: true,
  }
);
mongoose.Promise = global.Promise;
let db = mongoose.connection;
db.on("error", console.error.bind(console, "MongoDB connection error:"));

And Environment variable is as follows in deployment file.

env:
  - name: MONGO_CONNECTION_STRING
    value: "mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo:27017/test"

Mongo pods status

mongo-0                                         2/2     Running     0          8m35s
mongo-1                                         2/2     Running     0          7m49s
mongo-2                                         2/2     Running     0          6m54s

kubectl get statefulset

NAME     READY   AGE
mongo    3/3     9m31s

Service

NAME                                  TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
mongo                                 ClusterIP      None           <none>          27017/TCP                    27m

rs.config() output

{
	"_id" : "rs0",
	"version" : 1,
	"protocolVersion" : NumberLong(1),
	"members" : [
		{
			"_id" : 0,
			"host" : "mongo-0.mongo:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
				
			},
			"slaveDelay" : NumberLong(0),
			"votes" : 1
		},
		{
			"_id" : 1,
			"host" : "mongo-1.mongo:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
				
			},
			"slaveDelay" : NumberLong(0),
			"votes" : 1
		},
		{
			"_id" : 2,
			"host" : "mongo-2.mongo:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
				
			},
			"slaveDelay" : NumberLong(0),
			"votes" : 1
		}
	],
	"settings" : {
		"chainingAllowed" : true,
		"heartbeatIntervalMillis" : 2000,
		"heartbeatTimeoutSecs" : 10,
		"electionTimeoutMillis" : 10000,
		"catchUpTimeoutMillis" : 60000,
		"getLastErrorModes" : {
			
		},
		"getLastErrorDefaults" : {
			"w" : 1,
			"wtimeout" : 0
		},
		"replicaSetId" : ObjectId("5f16a0f3671c091fe183af72")
	}
}

Any help is appreciated.

-- Zain Saleem
kubernetes
mongodb
mongoose

1 Answer

7/21/2020

Figured it out

Tutorial has a step missing to initiate the set.

After connecting to set using the command

kubectl exec -ti mongo-0 mongo

Just Run following two commands

rs.initiate({_id: "rs0", version: 1, members: [
       { _id: 0, host : "mongo-0.mongo:27017" },
       { _id: 1, host : "mongo-1.mongo:27017" },
       { _id: 2, host : "mongo-2.mongo:27017" }
 ]});

rs.slaveOk()
-- Zain Saleem
Source: StackOverflow