I have a K8s cluster running on OpenStack. I'm deploying my app along with single node MongoDB 4.0.1 using helm and MongoDB helm chart 4.0.6. MongoDB seems to initialize and start just fine. However, after starting up, all authentication is failing. An odd twist to this is that it worked originally, but now it fails every time i try/retry.

MongoDB log for authn failure:

Welcome to the Bitnami mongodb container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues

nami    INFO  Initializing mongodb
mongodb INFO  ==> Deploying MongoDB from scratch...
mongodb INFO  ==> No injected configuration files found. Creating default config files...
mongodb INFO  ==> Creating root user...
mongodb INFO  ==> Creating ars user...
mongodb INFO  ==> Enabling authentication...
mongodb INFO 
mongodb INFO  
mongodb INFO   Installation parameters for mongodb:
mongodb INFO     Root Password: **********
mongodb INFO     Username: ars
mongodb INFO     Password: **********
mongodb INFO     Database: ars02
mongodb INFO   (Passwords are not shown for security reasons)
mongodb INFO  
mongodb INFO 
nami    INFO  mongodb successfully initialized
INFO  ==> Starting mongodb... 
INFO  ==> Starting mongod...

and

2018-10-11T17:44:39.192+0000 I ACCESS   [conn231] SASL SCRAM-SHA-1 authentication failed for ars on ars02 from client 10.100.49.5:37116 ; UserNotFound: Could not find user ars@ars02

Occasionally, i get an assert failure in the startup sequence:

2018-10-25T20:07:03.942+0000 F STORAGE  [initandlisten] Unable to start up mongod due to missing featureCompatibilityVersion document.
2018-10-25T20:07:03.942+0000 F STORAGE  [initandlisten] Please run with --repair to restore the document.
2018-10-25T20:07:03.942+0000 F -        [initandlisten] Fatal Assertion 40652 at src/mongo/db/repair_database_and_check_version.cpp 579
2018-10-25T20:07:03.942+0000 F -        [initandlisten] 

2. docker image environemnt:

   MONGODB_ROOT_PASSWORD=ThisIsTheMongoRootPassword MONGODB_PRIMARY_ROOT_USER=root MONGODB_PRIMARY_ROOT_PASSWORD= MONGODB_REPLICA_SET_MODE= MONGODB_ADVERTISED_HOSTNAME= MONGODB_PRIMARY_HOST= MONGODB_REPLICA_SET_NAME=replicaset MONGODB_DATABASE=ars02 MONGODB_PRIMARY_PORT_NUMBER=27017 MONGODB_EXTRA_FLAGS= MONGODB_PASSWORD=ars MONGODB_USERNAME=ars MONGODB_ENABLE_IPV6=yes MONGODB_REPLICA_SET_KEY=

3. Attempt to authenticate in mongo shell:

   $ mongo ars02 -u ars -p ars MongoDB shell version v4.0.1 connecting to: mongodb://127.0.0.1:27017/ars02 MongoDB server version: 4.0.1 2018-10-11T17:54:05.601+0000 E QUERY [js] Error: Authentication failed. : DB.prototype._authOrThrow@src/mongo/shell/db.js:1679:20 @(auth):6:1 @(auth):1:2 exception: login failed

   $ mongo admin -u root -p ThisIsTheMongoRootPassword MongoDB shell version v4.0.1 connecting to: mongodb://127.0.0.1:27017/ars02 MongoDB server version: 4.0.1 2018-10-11T17:54:32.645+0000 E QUERY [js] Error: Authentication failed. : DB.prototype._authOrThrow@src/mongo/shell/db.js:1679:20 @(auth):6:1 @(auth):1:2 exception: login failed

   $ mongo ars02 -u root -p ThisIsTheMongoRootPassword MongoDB shell version v4.0.1 connecting to: mongodb://127.0.0.1:27017/admin MongoDB server version: 4.0.1 2018-10-11T17:54:42.456+0000 E QUERY [js] Error: Authentication failed. : DB.prototype._authOrThrow@src/mongo/shell/db.js:1679:20 @(auth):6:1 @(auth):1:2 exception: login failed

4. associated ACCESS logs:

   2018-10-11T18:05:17.544+0000 I ACCESS [conn134] Supported SASL mechanisms requested for unknown user 'ars@ars02' 2018-10-11T18:05:17.544+0000 I ACCESS [conn134] SASL SCRAM-SHA-1 authentication failed for ars on ars02 from client 127.0.0.

   2018-10-11T18:02:43.542+0000 I ACCESS [conn50] Supported SASL mechanisms requested for unknown user 'root@admin' 2018-10-11T18:02:43.543+0000 I ACCESS [conn50] SASL SCRAM-SHA-1 authentication failed for root on admin from client 127.0.0.1:46832 ; UserNotFound: Could not find user root@admin

   2018-10-11T18:04:11.144+0000 I ACCESS [conn100] Supported SASL mechanisms requested for unknown user 'root@ars02' 2018-10-11T18:04:11.144+0000 I ACCESS [conn100] SASL SCRAM-SHA-1 authentication failed for root on ars02 from client 127.0.0

my loose understanding is that the initialization is driven off the environment variables set up by Kubernetes. there is a mongo-inputs.json that is referenced in the bitnami initialization scripts which seems to validate that:

$ cat mongodb-inputs.json 
{
  "advertisedHostname": "{{$global.env.MONGODB_ADVERTISED_HOSTNAME}}",
  "database": "{{$global.env.MONGODB_DATABASE}}",
  "enableIPv6": "{{$global.env.MONGODB_ENABLE_IPV6}}",
  "password": "{{$global.env.MONGODB_PASSWORD}}",
  "primaryHost": "{{$global.env.MONGODB_PRIMARY_HOST}}",
  "primaryPort": "{{$global.env.MONGODB_PRIMARY_PORT_NUMBER}}",
  "primaryRootPassword": "{{$global.env.MONGODB_PRIMARY_ROOT_PASSWORD}}",
  "primaryRootUser": "{{$global.env.MONGODB_PRIMARY_ROOT_USER}}",
  "replicaSetKey": "{{$global.env.MONGODB_REPLICA_SET_KEY}}",
  "replicaSetMode": "{{$global.env.MONGODB_REPLICA_SET_MODE}}",
  "replicaSetName": "{{$global.env.MONGODB_REPLICA_SET_NAME}}",
  "rootPassword": "{{$global.env.MONGODB_ROOT_PASSWORD}}",
  "username": "{{$global.env.MONGODB_USERNAME}}"
}

my MongoDB deployment is:

---
# Source: v/charts/mongodb/templates/deployment-standalone.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: v-test-mongodb
  labels:
    app: mongodb
    chart: mongodb-4.0.6
    release: "v-test"
    heritage: "Tiller"
spec:
  template:
    metadata:
      labels:
        app: mongodb
        release: "v-test"
        chart: mongodb-4.0.6
    spec:
      securityContext:
        fsGroup: 1001
        runAsUser: 1001
      containers:
      - name: v-test-mongodb
        image: docker.io/bitnami/mongodb:4.0.1-debian-9
        imagePullPolicy: "Always"
        env:
        - name: MONGODB_ROOT_PASSWORD
          value: "ThisIsTheMongoRootPassword"
        - name: MONGODB_USERNAME
          value: "ars"
        - name: MONGODB_PASSWORD
          value: "ars"
        - name: MONGODB_DATABASE
          value: "ars02"
        - name: MONGODB_EXTRA_FLAGS
          value: 
        ports:
        - name: mongodb
          containerPort: 27017
        livenessProbe:
          exec:
            command:
            - mongo
            - --eval
            - "db.adminCommand('ping')"
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 6
        readinessProbe:
          exec:
            command:
            - mongo
            - --eval
            - "db.adminCommand('ping')"
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 6
        volumeMounts:
        - name: data
          mountPath: /bitnami/mongodb
        resources:
          limits:
            cpu: 1
            memory: 6Gi
          requests:
            cpu: 100m
            memory: 1Gi

      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: v-test-mongodb
---

i switched from using MONGODB_DATABASE / MONGODB_USERNAME / MONGODB_PASSWORD to running an initialization script out of the docker-entrypoint-initdb.d directory. this seems to help, but i am still sometimes seeing both the authn and the assert failures. when authn fails my init script cannot connect to the admin database as root to create the user / database.

is anyone else seeing these kinds of issues deploying mongodb?