K
Q

Trying to connect to mongodb service through Consul Connect Sidecar Proxy

4/22/2020

I have a Minikube set up and a mongo instance running in it. I use Consul + Consul Connect to mesh my services. Only I can not connect to mongo from another service using sidecar upstreams, some weird stuff is happening...

My mongo instance is installed using bitnami helm chart, I just set the service name, set username and change the storage class to match my need, and put consul annotations for service mesh in pod annotation section:

image:
  registry: docker.io
  repository: bitnami/mongodb
  tag: 4.2.5-debian-10-r3
  pullPolicy: IfNotPresent
  debug: false
serviceAccount:
  create: true
  name: "svc-identity-data"
usePassword: true
mongodbRootPassword: rootpassword
mongodbUsername: identity
mongodbPassword: identity
mongodbDatabase: company
service:
  name: svc-identity-data
  annotations: {}
  type: ClusterIP
  port: 27017
useStatefulSet: true
replicaSet:
  enabled: false
  useHostnames: true
  name: rs0
  replicas:
    secondary: 1
    arbiter: 1
  pdb:
    enabled: true
    minAvailable:
      primary: 1
      secondary: 1
      arbiter: 1
annotations: {}
labels: {}
podAnnotations:
    "consul.hashicorp.com/connect-inject": "true"
    "consul.hashicorp.com/connect-service": "svc-identity-data"
    "consul.hashicorp.com/connect-service-protocol": "tcp"
persistence:
  enabled: true
  mountPath: /bitnami/mongodb
  subPath: ""
  storageClass: "standard"
  accessModes:
    - ReadWriteOnce
  size: 8Gi
  annotations: {}
configmap:
  storage:
    dbPath: /bitnami/mongodb/data/db
    journal:
      enabled: true
    directoryPerDB: false
  systemLog:
    destination: file
    quiet: false
    logAppend: true
    logRotate: reopen
    path: /opt/bitnami/mongodb/logs/mongodb.log
    verbosity: 0
  net:
    port: 27017
    unixDomainSocket:
      enabled: true
      pathPrefix: /opt/bitnami/mongodb/tmp
    ipv6: false
    bindIp: 0.0.0.0
  processManagement:
     fork: false
     pidFilePath: /opt/bitnami/mongodb/tmp/mongodb.pid
  setParameter:
     enableLocalhostAuthBypass: true
  security:
    authorization: enabled

Secondly I started a stand-alone mongodb pod to use mongo client, and meshed with consul connect using annotations

apiVersion: v1
kind: Pod
metadata:
  name: mongo-client
  labels:
    name: mongo-client
  annotations:
        "consul.hashicorp.com/connect-inject": "true"
        "consul.hashicorp.com/connect-service-upstreams": "svc-identity-data:28017"
        "consul.hashicorp.com/connect-service-protocol": "tcp"    
spec:
  containers:
  - name: mongo-client
    image: mongo:4.2.5
    imagePullPolicy: IfNotPresent
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    ports:
      - containerPort: 27017

I now have a mongodb service and a mongo client pod with an upstream to mongodb service binded on 127.0.0.1:28017

When I try to connect to mongodb service using my upstream I get a behavior I don't understand

> kubectl exec -it mongo-client mongo --host 127.0.0.1 --port 28017 -u root -p rootpassword

MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("8c46012d-8083-4029-8495-167bbe8bf063") }
MongoDB server version: 4.2.5
Server has startup warnings: 
2020-04-22T12:20:14.777+0000 I  STORAGE  [initandlisten] 
2020-04-22T12:20:14.777+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

> 
bye

No problem here, everything works perfectly fine to me, but if I use mongo with a connection string instead of separate parameters, I get a connection refused

> kubectl exec -it mongo-client mongo mongodb://root:roopassword@127.0.0.1:28017/?authSource=admin

MongoDB shell version v4.2.5
connecting to: mongodb://127.0.0.1:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-22T15:04:07.955+0000 I  NETWORK  [js] DBClientConnection failed to receive message from 127.0.0.1:28017 - HostUnreachable: Connection closed by peer
2020-04-22T15:04:07.968+0000 E  QUERY    [js] Error: network error while attempting to run command 'isMaster' on host '127.0.0.1:28017'  :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-04-22T15:04:07.973+0000 F  -        [main] exception: connect failed
2020-04-22T15:04:07.973+0000 E  -        [main] exiting with code 1

I don't understand at all what is the difference between using connection string and separate parameters, if you have any clue or a solution, please let me know.

P.S : I didn't set any secure communication (tls), I'm on a minikube (because I'm a Microservice Architecture and Kubernetes n00b) and it is to experiment service mesh (we need to live in the current era), a solution involving connecting to service without using the sidecar is not the point, by the way connecting directly to service works perfectly using connection string.

> kubectl exec -it mongo-client mongo -mongodb://root:roopassword@svc-identity-data:28017/?authSource=admin

MongoDB shell version v4.2.5
connecting to: mongodb://svc-identity-data:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("713febaf-2000-4ca6-8b1f-963c76986e72") }
MongoDB server version: 4.2.5
Server has startup warnings: 
2020-04-22T12:20:14.777+0000 I  STORAGE  [initandlisten] 
2020-04-22T12:20:14.777+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-22T12:20:14.777+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

> 
bye

EDIT : Rebooting minikube make all things work as intended. I will investigate more on the matter to understand why. Maybe someone else will hit the same issue.

EDIT 2 : I discovered one thing : connection error when connecting to mongo through sidecar is random, when I run command until it success, here is what I get

root@mongo-client:/# mongo mongodb://root:rootpassword@localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
2020-04-24T12:51:15.641+0000 I  NETWORK  [js] DBClientConnection failed to receive message from localhost:28017 - HostUnreachable: Connection closed by peer
2020-04-24T12:51:15.702+0000 E  QUERY    [js] Error: network error while attempting to run command 'isMaster' on host 'localhost:28017'  :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-04-24T12:51:15.729+0000 F  -        [main] exception: connect failed
2020-04-24T12:51:15.729+0000 E  -        [main] exiting with code 1
root@mongo-client:/# mongo mongodb://root:rootpassword@localhost:28017/?authSource=admin
MongoDB shell version v4.2.5
connecting to: mongodb://localhost:28017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("628bfcf9-6d44-4168-ab74-19a717d746f6") }
MongoDB server version: 4.2.5
Server has startup warnings: 
2020-04-24T06:43:39.359+0000 I  STORAGE  [initandlisten] 
2020-04-24T06:43:39.359+0000 I  STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-04-24T06:43:39.359+0000 I  STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

> 
bye

And on the side of mongo the log :

2020-04-24T12:51:19.281+0000 I  NETWORK  [conn6647] end connection 127.0.0.1:54148 (6 connections now open)
2020-04-24T12:51:19.526+0000 I  COMMAND  [conn6646] command admin.$cmd appName: "MongoDB Shell" command: saslStart { saslStart: 1, mechanism: "SCRAM-SHA-256", payload: "xxx", $db: "admin" } numYields:0 reslen:196 locks:{} protocol:op_msg 231ms
2020-04-24T12:51:19.938+0000 I  ACCESS   [conn6646] Successfully authenticated as principal root on admin from client 127.0.0.1:54142
2020-04-24T12:51:20.024+0000 I  NETWORK  [listener] connection accepted from 127.0.0.1:54168 #6648 (7 connections now open)
2020-04-24T12:51:20.027+0000 I  NETWORK  [conn6648] received client metadata from 127.0.0.1:54168 conn6648: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.5" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 10 (buster)"", architecture: "x86_64", version: "Kernel 4.19.94" } }
2020-04-24T12:51:20.215+0000 I  NETWORK  [conn6648] end connection 127.0.0.1:54168 (6 connections now open)
2020-04-24T12:51:21.328+0000 I  NETWORK  [conn6646] end connection 127.0.0.1:54142 (5 connections now open)

I am more and more confused, I can not explain that behavior.

-- SmasherHell
consul
envoyproxy
kubernetes
mongodb

Similar Questions

How do I expose a registry hosted in Minikube?
launch/scale a containerized app on http request
Occassional Unable to mount volumes for pod when mounting nfs persistent volume claim with ReadWriteMany permissions
Error: x509: certificate signed by unknown authority, kind cluster

2 Answers

4/24/2020

The problem may be that CN of the certificate doesn't match the value of hostname in config file of MongoDB. Its is about MongoDB specification and parameters with which you are running it.

CN (common name) or SAN (subject alternative name) of the certificate has to match the value of --hostname that you supply when running mongo. Your MongoDB URI is:

MONGODB_URI=mongodb://root:roopassword@127.0.0.1:28017/?authSource=admin

the MongoDB is NOT on localhost. Also the MongoDB server needs to allow ANY host to connect to the database. By default it will ONLY allow connections from the SAME runtime. You need to get IP address of service which is assigned to pod with your database container - svc-identity-data has address 10.107.99.51.

Take a look: mongodb-ssl, mongodb-failed-to-connect

-- MaggieO
Source: StackOverflow
4/24/2020

I found out the solution, it turns out to be the simpliest issue possible : resources

My minikube wasn't enough to make all pods running swiftly, it was introducing a latency between the sidecar proxy pods even if kubenetes raised no error on any outage.

I'm a kubernetes learner so I didn't think of it right away. Now that I know what happened I can investigate in the right direction to undestand in what extends the latency can be an issue.

-- SmasherHell
Source: StackOverflow