Connecting to sharded mongodb cluster through Java application from Kubernetes pod using istio

11/5/2018

I have a sharded mongodb cluster hosting on atlas on GCP https://cloud.mongodb.com

I can connect to it when I run my docker container on local by connecting to the primary shard myapp-shard-00-01-vki7g.gcp.mongodb.net:27017 through my java application by configuring it this manner without any problem.

java -Dlog4j.configurationFile=resources/log4j2.properties \
                 -Dmyapp.myapp.userDatabases=mydb \
                 -Dmyapp.myapp.port=27018 \
                 -Dmyapp.mongo.host=myapp-shard-00-01-vki7g.gcp.mongodb.net \
                 -Dmyapp.mongo.port=27017 \
-Dmyapp.mongo.sslEnabled=true \
     -Dmyapp.mongo.authenticationMechanism=SCRAM-SHA-1 \
                 -Xms1G \
                 -Xmx3G \
                 -XX:+UseParallelGC \
                 -XX:+HeapDumpOnOutOfMemoryError \
                 -XX:HeapDumpPath=logs/java_pid.hprof \
                 -XX:+UseGCOverheadLimit \
                 -server \
                 -XX:+UseStringDeduplication \
                 -jar myapp.jar

export ENV_MYAPP_MONGO_USER=myusername export ENV_MYAPP_MONGO_PWD=mypassword export ENV_MYAPP_MONGO_AUTH_SOURCE=admin

However, when I set up the same container on Kubernetes and run the same command along with setting up the environment variables, I get the following error

com.mongodb.MongoCommandException: Command failed with error 40413 (Location40413): 'BSON field 'OperationSessionInfo.$clusterTime' is a duplicate field' on server m8yapp-staging-shard-00-01-vki7g.gcp.mongodb.net:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1541424794, "i" : 3 } }, "ok" : 0.0, "errmsg" : "BSON field 'OperationSessionInfo.$clusterTime' is a duplicate field", "code" : 40413, "codeName" : "Location40413", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1541424794, "i" : 3 } }, "signature" : { "hash" : { "$binary" : "Lo375z7JMqIYZKjRhlXvJwQzoNE=", "$type" : "00" }, "keyId" : { "$numberLong" : "6592814596526964737" } } } }
    at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:179) ~[mongodb-driver-core-3.8.2.jar!/:?]
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:293) ~[mongodb-driver-core-3.8.2.jar!/:?]
    at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) [mongodb-driver-core-3.8.2.jar!/:?]
    at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) [mongodb-driver-core-3.8.2.jar!/:?]
    at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444) [mongodb-driver-core-3.8.2.jar!/:?]

I tried to setup the following egress in the following manner:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mongodb-staging
spec:
  hosts:
  - myapp-shard-00-01-vki7g.gcp.mongodb.net
  - myapp-shard-00-00-vki7g.gcp.mongodb.net
  - myapp-shard-00-02-vki7g.gcp.mongodb.net
  ports:
  - number: 27017
    name: mongo
    protocol: MONGO
  resolution: DNS

I also tries setting the protocol to TCP and resolution to NONE.

I can connect to this mongodb from local and also the kubernetes using the following command:

mongo --host myapp-shard-00-01-vki7g.gcp.mongodb.net --port 27017 --username <myusername> --password <mypassword> --ssl --authenticationDatabase admin

I can't post my Java code here, but I am not quite sure why the docker container passes when I run on my local while it throws the above error on Kubernetes.

EDIT

I tried adding this

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mongodb-staging
spec:
  hosts:
  - myapp-shard-00-01-vki7g.gcp.mongodb.net
  ports:
  - number: 27017
    name: tls-mongo
    protocol: tls
  resolution: DNS
  location: MESH_EXTERNAL

---  
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: mongodb
spec:
  hosts:
  - myapp-shard-00-01-vki7g.gcp.mongodb.net
  tls:
  - match:
    - port: 27017
      sni_hosts:
      - myapp-shard-00-01-vki7g.gcp.mongodb.net
    route:
    - destination:
        host: myapp-shard-00-01-vki7g.gcp.mongodb.net #primary
        port:
          number: 27017
      weight: 100

I added this and I still get the same error.

-- kosta
istio
kubernetes
mongodb

1 Answer

11/5/2018

You have to configure access to MongoDB as TCP with IP blocks in CIDR notation (see https://preliminary.istio.io/blog/2018/egress-tcp/) or as TLS with SNI (see https://preliminary.istio.io/docs/tasks/traffic-management/egress/, the www.google.com example, just replace HTTPS with TLS.

I actually completed a PR today about consuming external MongoDB services - see https://github.com/istio/istio.io/pull/2347, there I specify various options for controlling external traffic to MongoDB.

-- Vadim Eisenberg
Source: StackOverflow