The mongodb agent never gets healthy when I increase the replica set. If I initially start with 3 members, all 3 members get created successfully. If I decrease, it also works (after a while).

Steps to reproduce: 1. Installed mongodb-community-operator version 0.7.0 clusterwide. 1. Create a mongodb community object with 1 member (see code below). Tested with versions 4.4.0 and 5.0.0 1. wait until pods are running. 1. increase members 1. mongodb-agent from first member never gets healthy. Second member won't be created.

Findings:

• According to point 4 in https://github.com/mongodb/mongodb-kubernetes-operator/blob/98b3c78aaca0c58b951156e6f4d7770fd913ebb6/docs/deploy-configure.md#deploy-a-replica-set the operator should create a secret containing connection strings. That's not the case.
• The mongodb community object does not get a status, although it should have. Found this error in the mongodb operator pod logs: Error updating the status of the MongoDB resource: MongoDBCommunity.mongodbcommunity.mongodb.com "my-database" is invalid: status.version: Required value

• Found a log in the mongodb-agent container, which indicates that the missing populated status field leads to a step "StartFresh" instead of "Start" at reconcilation:

  2021-10-05T08:21:04.124+0000 src/director/director.go:tracef:794 0> 08:21:04.124 Running step: 'StartFresh' of move 'Start' 2021-10-05T08:21:04.124+0000 src/director/director.go:tracef:794 0> 08:21:04.124 because [All the following are true: 'currentState.Running' = false 'desiredState.ProcessType' != mongos ('desiredState.ProcessType' = mongod) All the following are true: ['currentState.BackupRestoreFinished' = true] ['currentState.IsDirectAttachRestoreRequested' = false] ]

I also checked, deploying all pods on one node, to mitigate subnet connection issues. No change. The only (working) way to increase the member count is to delete all pvcs and start a new deployment with the increased member count. But the data is deleted then, so that's not really a workaround.

Here my code for the mongodbcommunity object:

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: my-database
  namespace: my-namespace
spec:
  members: 3
  type: ReplicaSet
  version: "5.0.0"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-database-user
      db: admin
      passwordSecretRef:
        name: my-database-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
        - name: readWrite
          db: my-database
      scramCredentialsSecretName: my-scram
  additionalMongodConfig:
    storage.wiredTiger.engineConfig.journalCompressor: zlib