Please see edit below for added information.

So long story short I'm trying to get the Unifi Controller to run on my home Kubernetes cluster. In doing so, I have needed to decentralize the MongoDB database since having a MongoDB instance bundled with each replica on Kubernetes causes the database to crash. Here is my project this far: https://github.com/zimmertr/Kubernetes-Manifests/tree/unifi_mongodb_separation/Unifi_Controller

In doing so, I have written the following script that runs at provision time for the MongoDB container:

mongo \
    --username ubnt \
    --password "{{ mongodb_password }}" \
    --authenticationDatabase admin \
    --eval 'db.getSiblingDB("unifi").createUser({user: "ubnt", pwd: "{{ mongodb_password }}", roles: [{role: "readWrite", db: "unifi"}]})'

mongo \
    --username ubnt \
    --password "{{ mongodb_password }}" \
    --authenticationDatabase admin \
    --eval 'db.getSiblingDB("unifi_stat").createUser({user: "ubnt", pwd: "{{ mongodb_password }}", roles: [{role: "readWrite", db: "unifi_stat"}]})'

And then I have configured the Unifi Controller to talk to the database through a volume mounted system.properties file configured like so:

# Inform IP Address
system_ip={{ load_balancer_ip }}

# Autobackup directory
autobackup.dir=/backups

# External MongoDB information
db.mongo.local=false
db.mongo.uri=mongodb://ubnt:{{ mongodb_password }}@unifi-controller-mongodb:27017/unifi
statdb.mongo.uri=mongodb://ubnt:{{ mongodb_password }}@unifi-controller-mongodb:27017/unifi_stat
unifi.db.name=unifi

This is configured as instructed by Ubiquiti.

This all works, and when the Kubernetes deployments start up I see that the Unifi Controller connects to the MongoDB instance in the logs. Furthermore, if I manually connect to the MongoDB databases and run a show collections I can see that many new collections have been created. However, the Unifi Controller stops producing logs here.

If I manually stop the jar file that is running the Unifi Controller in the background on the container, and then restart it, the following stack trace is produced:

gt; s6-setuidgid abc java -Xmx1024M -jar /usr/lib/unifi/lib/ace.jar start
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called
Exception in thread "launcher" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'Ò00000' defined in class com.ubnt.service.AppContext: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [com.ubnt.service.P.D com.ubnt.service.AppContext.Ò00000()] threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbService' defined in class com.ubnt.service.AppContext: Invocation of init method failed; nested exception is com.mongodb.CommandFailureException: { "serverUsed" : "unifi-controller-mongodb:27017" , "ok" : 0.0 , "errmsg" : "not authorized on unifi to execute command { dropDatabase: 1 }" , "code" : 13 , "codeName" : "Unauthorized"}

The key element here is

not authorized on unifi to execute command { dropDatabase: 1 }"

And this is where my understanding of MongoDB comes to an end. And my question comes to a beginning. I believe that the reason the Unifi Controller does not continue to start or log any additional messages after connecting to the database and creating the collections is that it is silently failing to perform an action on the MongoDB database. For which it does not have the requisite permissions to do.

When I provision the MongoDB Docker container, I specify the MONGO_INITDB_ROOT_USERNAME & MONGO_INITDB_ROOT_PASSWORD environment variables. Which, I believe, turns auth mode on. This belief is reiterated by the fact that I can connect to MongoDB's admin authentication database via the username and password I provide to these variables.

However, based on the script I posted above that creates my databases and assigns the role readWrite to the ubnt user, I'm curious how I should go about giving the ubnt user the requisite permissions required to drop a database. If I swap out readWrite for other roles like root and dbAdminAnyDatabase the commands fail.

What do I have to do to make it so my ubnt user can drop the unifi and unifi_stat databases? Or what do I have to change about my connection strings to prevent this from happening? I'm a bit of a database admin noob.

Continuation Edit:

I have updated the Role that is attributed to the ubnt user on unifi and unifi_stat to be dbAdmin instead of readWrite. And got a little further.

#!/bin/bash
mongo \
    --username ubnt \
    --password "{{ mongodb_password }}" \
    --authenticationDatabase admin \
    --eval 'db.createUser({user: "ubnt", pwd: "{{ mongodb_password }}", roles: [{role: "dbAdmin", db: "unifi"}]})'

mongo \
    --username ubnt \
    --password "{{ mongodb_password }}" \
    --authenticationDatabase admin \
    --eval 'db.createUser({user: "ubnt", pwd: "{{ mongodb_password }}", roles: [{role: "dbAdmin", db: "unifi_stat"}]})'

However, the Unifi Controller is still acting strange. It is now simply looping this in the log files:

2019-02-10 22:33:45,449] <launcher> INFO  system - ======================================================================
[2019-02-10 22:33:45,450] <launcher> INFO  system - UniFi 5.6.40 (build atag_5.6.40_10370 - release) is started
[2019-02-10 22:33:45,450] <launcher> INFO  system - ======================================================================
[2019-02-10 22:33:45,457] <launcher> INFO  system - BASE dir:/usr/lib/unifi
[2019-02-10 22:33:45,464] <launcher> INFO  system - Current System IP: 192.168.0.1
[2019-02-10 22:33:45,465] <launcher> INFO  system - Hostname: unifi-controller-5bb95c7688-bzp4z
[2019-02-10 22:33:48,635] <launcher> INFO  db     - waiting for db connection...
[2019-02-10 22:33:49,173] <launcher> INFO  db     - Connecting to mongodb://ubnt:PASSWORD@unifi-controller-mongodb:27017/unifi
[2019-02-10 22:33:49,526] <launcher> DEBUG db     - db connected (3.4.19@unifi-controller-mongodb:27017)
[2019-02-10 22:33:49,534] <launcher> INFO  db     - *** Factory Default *** Database exists. Drop it
[2019-02-10 22:33:52,391] <launcher> INFO  db     - waiting for db connection...
[2019-02-10 22:33:52,896] <launcher> DEBUG db     - db connected (3.4.19@unifi-controller-mongodb:27017)
[2019-02-10 22:34:13,292] <launcher> INFO  system - ======================================================================
[2019-02-10 22:34:13,295] <launcher> INFO  system - UniFi 5.6.40 (build atag_5.6.40_10370 - release) is started
[2019-02-10 22:34:13,295] <launcher> INFO  system - ======================================================================
[2019-02-10 22:34:13,303] <launcher> INFO  system - BASE dir:/usr/lib/unifi
[2019-02-10 22:34:13,312] <launcher> INFO  system - Current System IP: 192.168.0.1
[2019-02-10 22:34:13,313] <launcher> INFO  system - Hostname: unifi-controller-5bb95c7688-bzp4z
[2019-02-10 22:34:16,781] <launcher> INFO  db     - waiting for db connection...
[2019-02-10 22:34:17,300] <launcher> INFO  db     - Connecting to mongodb://ubnt:PASSWORD@unifi-controller-mongodb:27017/unifi
[2019-02-10 22:34:17,640] <launcher> DEBUG db     - db connected (3.4.19@unifi-controller-mongodb:27017)
[2019-02-10 22:34:17,656] <launcher> INFO  db     - *** Factory Default *** Database exists. Drop it
[2019-02-10 22:34:20,463] <launcher> INFO  db     - waiting for db connection...
[2019-02-10 22:34:20,969] <launcher> DEBUG db     - db connected (3.4.19@unifi-controller-mongodb:27017)

So I'm at a loss here. Not sure why it's trying to drop the database? Is Unifi trying to create the databases from scratch on the MongoDB instance?

1. If I DON'T create the unifi and unifi_stat databases at provision time for MongoDB, then the Unifi Controller fails to ever connect to them and the logs stall there.

2. If I DO create the databases and give the ubnt user dbAdmin over them, then Unifi just appears to drop them over and over. Shown above.

3. If I DO create the databases and give the ubnt user readWrite over them, then Unifi just connects to the databases, creates the collections, and silently stalls for no apparent reason. And If I try and manually execute the jar file mentioned above then it leaves the stacktrace describing the lack of necessary permissions to drop a database.

If someone could please provide some documentation on how an external MongoDB database should be prepared for the Unifi Controller to use it would be greatly beneficial for me. I've only been able to track down a forum post  whereby an employee discussed how to configure the controller's system.properties to point to an external instance.