I am attempting to deploy 2 MongoDB instances; 1 to the staging namespace and 1 to the production namespace. My staging deployment is running fine, however the production data-retriever deployment crashes with the error MongoNetworkError: getaddrinfo ENOTFOUND

I am sure this is something that has been achieved before but I cannot find any documentation on achieving the desired outcome above.

The following YAML files are being used and the key differences are in the serviceName change in statefulset-prod.yml and MONGODB_URI in the environment variables of deployment-prod.yml:

service-staging.yml

apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: staging
  labels:
    app: my-app
    environment: staging
spec:
  ports:
  - name: http
    protocol: TCP
    port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongodb
    environment: staging

service-prod.yml

apiVersion: v1
kind: Service
metadata:
  name: mongodb-production
  namespace: production
  labels:
    app: my-app
    environment: production
spec:
  ports:
  - name: http
    protocol: TCP
    port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongodb-production
    environment: production

statefulset-staging.yml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongodb-staging
  namespace: staging
  labels:
    app: my-app
    environment: staging
  annotations:
        prometheus.io.scrape: "true"
spec:
  serviceName: "mongodb"
  replicas: 1
  template:
    metadata:
      labels:
        role: mongodb
        environment: staging
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
            - "--bind_ip_all"
            - "--wiredTigerCacheSizeGB=0.5"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongodb,environment=staging"
            - name: KUBERNETES_MONGO_SERVICE_NAME
              value: "mongodb"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast-storage
      resources:
        requests:
          storage: 25Gi

statefulset-prod.yml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongodb-production
  namespace: production
  labels:
    app: my-app
    environment: production
  annotations:
        prometheus.io.scrape: "true"
spec:
  serviceName: "mongodb-production"
  replicas: 1
  template:
    metadata:
      labels:
        role: mongodb-production
        environment: production
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
            - "--bind_ip_all"
            - "--wiredTigerCacheSizeGB=0.5"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongodb-production,environment=production"
            - name: KUBERNETES_MONGO_SERVICE_NAME
              value: "mongodb-production"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast-storage
      resources:
        requests:
          storage: 25Gi

deployment-staging.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: data-retriever-deployment
  namespace: staging
  labels:
    app: data-retriever
    environment: staging
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: data-retriever
        environment: staging
        tier: backend
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - data-retriever
            topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
        - name: gitlab-reg
      containers:
      - name: data-retriever
        image: registry.gitlab.com/xxx/xxx:latest
        env:
        - name: MONGODB_URI
          value: "mongodb://mongodb:27017/appdb_staging?replicaSet=rs0"
        - name: NODE_ENV
          value: "staging"

deployment-prod.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: data-retriever-deployment
  namespace: production
  labels:
    app: data-retriever
    environment: production
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: data-retriever
        environment: production
        tier: backend
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - data-retriever
            topologyKey: "kubernetes.io/hostname"
      imagePullSecrets:
        - name: gitlab-reg
      containers:
      - name: data-retriever
        image: registry.gitlab.com/xxx/xxx:latest
        env:
        - name: MONGODB_URI
          value: "mongodb://mongodb-production:27017/appdb_production?replicaSet=rs0"
        - name: NODE_ENV
          value: "production"

Logs

Unable to connect to database: MongoNetworkError: failed to connect to server [mongodb:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongodb mongodb:27017]

Unable to connect to database: MongoNetworkError: failed to connect to server [mongodb:27017] on first connect [MongoNetworkError: getaddrinfo ENOTFOUND mongodb mongodb:27017]
    at NativeConnection.db.on (/app/services/mongoose.service.js:25:19)
    at emitOne (events.js:116:13)
    at NativeConnection.emit (events.js:211:7)
    at process.nextTick (/node_modules/mongoose/lib/connection.js:575:37)
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickCallback (internal/process/next_tick.js:181:9)