K
Q

Nodejs Mongodb returns EHOSTUNREACH when requested from within a docker container hosted on kubernetes

12/2/2019

I've been struggling with this issue for a while now. This is probably my second or third request on stackoverflow, so if I forgot to mention something important, please let me know.

First things first, here are some information about the setup:

  • All Applications are hosted inside a kubernetes cluster
  • Nginx acts as a loadbalancer(not inside the kubernetes cluster)
  • The services communicate with each other over dns (e.g. .default.svc.cluster.local)
  • All services are reachable via dns name and port(There is a kubernetes service resource for each service)

The problem goes like this: Whenever my Webapplication tries to access the Service from the internet, the service responds with a HTTP 500 internal server error. When I look into the logs, the error says this:

{ MongoNetworkError: failed to connect to server [172.16.62.2:8635] on first connect [MongoNetworkError: connect EHOSTUNREACH 172.16.62.2:8635]
    at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:433:11)
    at Pool.emit (events.js:189:13)
    at createConnection (/node_modules/mongodb/lib/core/connection/pool.js:571:14)
    at connect (/node_modules/mongodb/lib/core/connection/pool.js:1008:9)
    at makeConnection (/node_modules/mongodb/lib/core/connection/connect.js:40:11)
    at callback (/node_modules/mongodb/lib/core/connection/connect.js:262:5)
    at Socket.err (/node_modules/mongodb/lib/core/connection/connect.js:287:7)
    at Object.onceWrapper (events.js:277:13)
    at Socket.emit (events.js:189:13)
    at emitErrorNT (internal/streams/destroy.js:82:8)
  name: 'MongoNetworkError',
  [Symbol(mongoErrorContextSymbol)]: {} }
{ MongoNetworkError: failed to connect to server [172.16.62.2:8635] on first connect [MongoNetworkError: connect EHOSTUNREACH 172.16.62.2:8635]
    at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:433:11)
    at Pool.emit (events.js:189:13)
    at createConnection (/node_modules/mongodb/lib/core/connection/pool.js:571:14)
    at connect (/node_modules/mongodb/lib/core/connection/pool.js:1008:9)
    at makeConnection (/node_modules/mongodb/lib/core/connection/connect.js:40:11)
    at callback (/node_modules/mongodb/lib/core/connection/connect.js:262:5)
    at Socket.err (/node_modules/mongodb/lib/core/connection/connect.js:287:7)
    at Object.onceWrapper (events.js:277:13)
    at Socket.emit (events.js:189:13)
    at emitErrorNT (internal/streams/destroy.js:82:8)
  name: 'MongoNetworkError',
  [Symbol(mongoErrorContextSymbol)]: {} }

My configuration file looks looks like this:

const MongoClient = require('mongodb').MongoClient;

// Connection URL
const url = 'mongodb://mongo-service.default.svc.cluster.local:27017';

let mongoClient;

/**
 * creates mongo client
 * @param {string} dbName name of the database you want to connect to
 * @return {Promise<unknown>} mongo client
 */
const _getMongoDB = (dbName = 'oli') => {
  return new Promise((resolve, reject) => {
    if (mongoClient) {
      resolve(mongoClient);
    } else {
      MongoClient.connect(url, {useNewUrlParser: true}).then(client => {
        console.log('Connected successfully to server');
        mongoClient = client.db(dbName);
        resolve(mongoClient);
      }).catch(err => {
        console.log(err);
        reject(err);
      });
    }
  });
};

const mongoDB = {
  getClient: _getMongoDB,
};

module.exports = mongoDB;

The weird part is, that when I use the mongo shell from inside the container, everything works fine. I just need to remove the preceding "mongodb://". I've already tried that in the service, without success.

I'd really appreciate any help or hint into a direction. Maybe someone has encountered a similiar problem.

EDIT: I've discovered, that the logs of the mongo-service are full of errors, except when I connected to it from the shell:

2019-12-03T08:27:24.177+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:27:24.205+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:27:24.600+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:27:25.587+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:27:26.036+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:27:26.441+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:27:26.962+0000 W  NETWORK  [listener] Error accepting new connection SocketException: remote_endpoint: Transport endpoint is not connected
2019-12-03T08:31:40.862+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54830 #512206 (1 connection now open)
2019-12-03T08:31:40.863+0000 I  NETWORK  [conn512206] Error receiving request from client: ProtocolError: Client sent an HTTP request over a native MongoDB connection. Ending connection from 172.16.5.1:54830 (connection id: 512206)
2019-12-03T08:31:40.863+0000 I  NETWORK  [conn512206] end connection 172.16.5.1:54830 (0 connections now open)
2019-12-03T08:31:51.356+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54832 #512207 (1 connection now open)
2019-12-03T08:32:25.055+0000 I  NETWORK  [conn512207] end connection 172.16.5.1:54832 (0 connections now open)
2019-12-03T08:32:31.034+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54834 #512208 (1 connection now open)
2019-12-03T08:33:33.251+0000 I  NETWORK  [conn512208] Error receiving request from client: SSLHandshakeFailed: SSL handshake received but server is started without SSL support. Ending connection from 172.16.5.1:54834 (connection id: 512208)
2019-12-03T08:33:33.251+0000 I  NETWORK  [conn512208] end connection 172.16.5.1:54834 (0 connections now open)
2019-12-03T08:35:18.704+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54844 #512209 (1 connection now open)
2019-12-03T08:35:18.826+0000 E  -        [conn512209] Assertion: Location34348: cannot translate opcode 2010 src/mongo/rpc/message.h 120
2019-12-03T08:35:18.859+0000 I  NETWORK  [conn512209] DBException handling request, closing client connection: Location34348: cannot translate opcode 2010
2019-12-03T08:35:18.859+0000 I  NETWORK  [conn512209] end connection 172.16.5.1:54844 (0 connections now open)
2019-12-03T08:35:33.191+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54846 #512210 (1 connection now open)
2019-12-03T08:35:33.192+0000 E  -        [conn512210] Assertion: Location34348: cannot translate opcode 2010 src/mongo/rpc/message.h 120
2019-12-03T08:35:33.192+0000 I  NETWORK  [conn512210] DBException handling request, closing client connection: Location34348: cannot translate opcode 2010
2019-12-03T08:35:33.192+0000 I  NETWORK  [conn512210] end connection 172.16.5.1:54846 (0 connections now open)
2019-12-03T08:35:39.370+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54848 #512211 (1 connection now open)
2019-12-03T08:35:39.371+0000 E  -        [conn512211] Assertion: Location34348: cannot translate opcode 2010 src/mongo/rpc/message.h 120
2019-12-03T08:35:39.371+0000 I  NETWORK  [conn512211] DBException handling request, closing client connection: Location34348: cannot translate opcode 2010
2019-12-03T08:35:39.371+0000 I  NETWORK  [conn512211] end connection 172.16.5.1:54848 (0 connections now open)
2019-12-03T08:38:01.610+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54850 #512212 (1 connection now open)
2019-12-03T08:38:01.611+0000 E  -        [conn512212] Assertion: Location34348: cannot translate opcode 2010 src/mongo/rpc/message.h 120
2019-12-03T08:38:01.612+0000 I  NETWORK  [conn512212] DBException handling request, closing client connection: Location34348: cannot translate opcode 2010
2019-12-03T08:38:01.612+0000 I  NETWORK  [conn512212] end connection 172.16.5.1:54850 (0 connections now open)
2019-12-03T08:38:15.269+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54852 #512213 (1 connection now open)
2019-12-03T08:38:15.270+0000 E  -        [conn512213] Assertion: Location34348: cannot translate opcode 2010 src/mongo/rpc/message.h 120
2019-12-03T08:38:15.270+0000 I  NETWORK  [conn512213] DBException handling request, closing client connection: Location34348: cannot translate opcode 2010
2019-12-03T08:38:15.270+0000 I  NETWORK  [conn512213] end connection 172.16.5.1:54852 (0 connections now open)
2019-12-03T08:41:17.804+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54856 #512214 (1 connection now open)
2019-12-03T08:41:17.804+0000 I  NETWORK  [conn512214] received client metadata from 172.16.5.1:54856 conn512214: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.1" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } }
2019-12-03T08:41:21.328+0000 I  NETWORK  [conn512214] end connection 172.16.5.1:54856 (0 connections now open)
2019-12-03T08:42:02.199+0000 I  NETWORK  [listener] connection accepted from 172.16.5.1:54858 #512215 (1 connection now open)
2019-12-03T08:42:02.199+0000 I  NETWORK  [conn512215] received client metadata from 172.16.5.1:54858 conn512215: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.1" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 9 (stretch)"", architecture: "x86_64", version: "Kernel 4.9.0-8-amd64" } }
2019-12-03T08:42:15.324+0000 I  NETWORK  [conn512215] end connection 172.16.5.1:54858 (0 connections now open)

Apparently this can happen, when the pod is behind a loadbalancer. After changing that, the error log stays clean. However, this doesn't solve the original problem.

-- lance dörksen
docker
kubernetes
mongodb
node.js

Similar Questions

Access parent (chart) context from subcharts
How can I connect with a database deployed at Kubernetes
Cert-Manager provide own SSL Certificate for AKS
Kubernetes Ingress SSL certificate
Creating new CRD resource with cluster-admin role
IBM-MQ chart create queues

1 Answer

12/4/2019

It turns out, that the problem wasn't with connectivity or anything. While debugging, I created a connectivity test script and executed it via node -e. Connection successful. So it had to be something different...

Since I'm not developing the microservice (just responsible for the infrastucture part), I didn't know that there were more than one connection strings in different files. (First time searching through the files didn't give me that knowledge). The older connection string was pointing to a long gone mongodb. Everything works fine now. Thanks @Oles Rid for helping me narrowing down the problem.

-- lance dörksen
Source: StackOverflow