I've found cause and solution! In my setup, I was sharing a directory over a network using NFS. This way, all my cluster nodes (minions) had access to common directory located at /mnt/nfs/data/.

Cause

The reason that mongo couldn't start was due to invalid Persistent Volumes. Namely, I was using persistent volume HostPath type - this will work for a single node testing, or if you manually create directory structure on all your cluster nodes e.g. /tmp/your_pod_data_dir/. But, if you will try to mount nfs directory as a hostPath it will cause problems -such I had!

Solution

For directories that are shared over Network File System use NFS persistent volume type (NFS Example)! Below you will find my setup and two solutions.

Setup

/etc/hosts - my cluster nodes.

# Cluster nodes
192.168.123.130 master
192.168.123.131 web01
192.168.123.132 compute01
192.168.123.133 compute02

List of exported NFS directories.

[vagrant@master]$ showmount -e
Export list for master:
/nfs/data compute*,web*
/nfs/www  compute*,web*

First solution

This solution, shows deployment that mounts nfs directory via volumes -have a look at volumes and volumeMounts section.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mongo
  labels:
    run: mongo
spec:
  selector:
    matchLabels:
      run: mongo
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        run: mongo
    spec:
      containers:
      - image: mongo:4.2.0-bionic
        name: mongo
        ports:
        - containerPort: 27017
          name: mongo
        volumeMounts:
        - name: phenex-nfs
          mountPath: /data/db
      volumes:
      - name: phenex-nfs
        nfs:
          # IP of master node
          server: 192.168.123.130
          path: /nfs/data/phenex/production/permastore/mongo

Second solution

This solution, shows deployment that mounts nfs directory via volume claims -have a look at persistentVolumeClaim, Persistent Volume and Persistent Volume Claim are defined below.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mongo
  labels:
    run: mongo
spec:
  selector:
    matchLabels:
      run: mongo
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        run: mongo
    spec:
      containers:
      - image: mongo:4.2.0-bionic
        name: mongo
        ports:
        - containerPort: 27017
          name: mongo
        volumeMounts:
        - name: phenex-nfs
          mountPath: /data/db
      volumes:
      - name: phenex-nfs
        persistentVolumeClaim:
          claimName: phenex-nfs

Persistent Volume - NFS

apiVersion: v1
kind: PersistentVolume
metadata:
  name: phenex-nfs
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  nfs:
    # IP of master node
    server: 192.168.123.130
    path: /nfs/data
  claimRef:
    name: phenex-nfs
  persistentVolumeReclaimPolicy: Retain

Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: phenex-nfs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi

Expected Output

# Checking cluster state
[vagrant@master ~]$ kubectl get deploy,po,pv,pvc --output=wide
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES               SELECTOR
deployment.extensions/mongo   1/1     1            1           18s   mongo        mongo:4.2.0-bionic   run=mongo

NAME                         READY   STATUS    RESTARTS   AGE   IP          NODE    NOMINATED NODE   READINESS GATES
pod/mongo-65b7d6fb9f-mcmvj   1/1     Running   0          18s   10.44.0.2   web01   <none>           <none>

NAME                            CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE     VOLUMEMODE
persistentvolume/phenex-nfs     1Gi        RWO            Retain           Bound    /phenex-nfs                             27s     Filesystem

NAME                                 STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE     VOLUMEMODE
persistentvolumeclaim/phenex-nfs     Bound    phenex-nfs     1Gi        RWO                           27s     Filesystem

# Attaching to pod and checking network bindings
[vagrant@master ~]$ kubectl exec -it mongo-65b7d6fb9f-mcmvj -- bash
root@mongo-65b7d6fb9f-mcmvj:/$ apt update
root@mongo-65b7d6fb9f-mcmvj:/$ apt install net-tools
root@mongo-65b7d6fb9f-mcmvj:/$ netstat -tunlp tcp 0 0 0.0.0.0:27017
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:27017           0.0.0.0:*               LISTEN      - 

# Running mongo clinet
root@mongo-65b7d6fb9f-mcmvj:/$ mongo
MongoDB shell version v4.2.0
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("45287a0e-7d41-4484-a267-5101bd20fad3") }
MongoDB server version: 4.2.0
Server has startup warnings: 
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] 
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] 
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] 
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] **        We suggest setting it to 'never'
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] 
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] **        We suggest setting it to 'never'
2019-08-14T18:03:29.703+0000 I  CONTROL  [initandlisten] 
---
Enable MongoDB's free cloud-based monitoring service, which will then receive and display
metrics about your deployment (disk utilization, CPU, operation statistics, etc).

The monitoring data will be available on a MongoDB website with a unique URL accessible to you
and anyone you share the URL with. MongoDB may use this information to make product
improvements and to suggest MongoDB products and deployment options to you.

To enable free monitoring, run the following command: db.enableFreeMonitoring()
To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---

>