Error mounting volumes on Openshift (Next gen)

11/15/2016

I'm testing the new Openshift platform based on Docker and Kubernetes.

I've created a new project from scratch, then when I try to deploy a simple MongoDB service (as well with a python app), I got the following errors in the Monitoring section in Web console:

Unable to mount volumes for pod "mongodb-1-sfg8t_rob1(e9e53040-ab59-11e6-a64c-0e3d364e19a5)": timeout expired waiting for volumes to attach/mount for pod "mongodb-1-sfg8t"/"rob1". list of unattached/unmounted volumes=[mongodb-data]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "mongodb-1-sfg8t"/"rob1". list of unattached/unmounted volumes=[mongodb-data]

It seems a problem mounting the PVC in the container, however the PVC is correctly created and bounded:

oc get pvc

Returns:

NAME           STATUS    VOLUME         CAPACITY   ACCESSMODES   AGE
mongodb-data   Bound     pv-aws-9dged   1Gi        RWO           29m

I've deployed it with the following commands:

oc process -f openshift/templates/mongodb.json | oc create -f -

oc deploy mongodb --latest

The complete log from Web console:

enter image description here

The content of the template that I used is:

{
    "kind": "Template",
    "apiVersion": "v1",
    "metadata": {
        "name": "mongo-example",
        "annotations": {
            "openshift.io/display-name": "Mongo example",
            "tags": "quickstart,mongo"
        }
    },
    "labels": {
        "template": "mongo-example"
    },
    "message": "The following service(s) have been created in your project: ${NAME}.",
    "objects": [
        {
            "kind": "PersistentVolumeClaim",
            "apiVersion": "v1",
            "metadata": {
                "name": "${DATABASE_DATA_VOLUME}"
            },
            "spec": {
                "accessModes": [
                    "ReadWriteOnce"
                ],
                "resources": {
                    "requests": {
                        "storage": "${DB_VOLUME_CAPACITY}"
                    }
                }
            }
        },
        {
            "kind": "Service",
            "apiVersion": "v1",
            "metadata": {
                "name": "${DATABASE_SERVICE_NAME}",
                "annotations": {
                    "description": "Exposes the database server"
                }
            },
            "spec": {
                "ports": [
                    {
                        "name": "mongodb",
                        "port": 27017,
                        "targetPort": 27017
                    }
                ],
                "selector": {
                    "name": "${DATABASE_SERVICE_NAME}"
                }
            }
        },
        {
            "kind": "DeploymentConfig",
            "apiVersion": "v1",
            "metadata": {
                "name": "${DATABASE_SERVICE_NAME}",
                "annotations": {
                    "description": "Defines how to deploy the database"
                }
            },
            "spec": {
                "strategy": {
                    "type": "Recreate"
                },
                "triggers": [
                    {
                        "type": "ImageChange",
                        "imageChangeParams": {
                            "automatic": true,
                            "containerNames": [
                                "mymongodb"
                            ],
                            "from": {
                                "kind": "ImageStreamTag",
                                "namespace": "",
                                "name": "mongo:latest"
                            }
                        }
                    },
                    {
                        "type": "ConfigChange"
                    }
                ],
                "replicas": 1,
                "selector": {
                    "name": "${DATABASE_SERVICE_NAME}"
                },
                "template": {
                    "metadata": {
                        "name": "${DATABASE_SERVICE_NAME}",
                        "labels": {
                            "name": "${DATABASE_SERVICE_NAME}"
                        }
                    },
                    "spec": {
                        "volumes": [
                            {
                                "name": "${DATABASE_DATA_VOLUME}",
                                "persistentVolumeClaim": {
                                    "claimName": "${DATABASE_DATA_VOLUME}"
                                }
                            }
                        ],
                        "containers": [
                            {
                                "name": "mymongodb",
                                "image": "mongo:latest",
                                "ports": [
                                    {
                                        "containerPort": 27017
                                    }
                                ],
                                "env": [
                                    {
                                        "name": "MONGODB_USER",
                                        "value": "${DATABASE_USER}"
                                    },
                                    {
                                        "name": "MONGODB_PASSWORD",
                                        "value": "${DATABASE_PASSWORD}"
                                    },
                                    {
                                        "name": "MONGODB_DATABASE",
                                        "value": "${DATABASE_NAME}"
                                    }
                                ],
                                "volumeMounts": [
                                    {
                                        "name": "${DATABASE_DATA_VOLUME}",
                                        "mountPath": "/data/db"
                                    }
                                ],
                                "readinessProbe": {
                                    "timeoutSeconds": 1,
                                    "initialDelaySeconds": 5,
                                    "exec": {
                                        "command": [ "/bin/bash", "-c", "mongo --eval 'db.getName()'"]
                                    }
                                },
                                "livenessProbe": {
                                    "timeoutSeconds": 1,
                                    "initialDelaySeconds": 30,
                                    "tcpSocket": {
                                        "port": 27017
                                    }
                                },
                                "resources": {
                                    "limits": {
                                        "memory": "${MEMORY_MONGODB_LIMIT}"
                                    }
                                }
                            }
                        ]
                    }
                }
            }
        }
    ],
    "parameters": [
        {
            "name": "NAME",
            "displayName": "Name",
            "description": "The name",
            "required": true,
            "value": "mongo-example"
        },    
        {
            "name": "MEMORY_MONGODB_LIMIT",
            "displayName": "Memory Limit (MONGODB)",
            "required": true,
            "description": "Maximum amount of memory the MONGODB container can use.",
            "value": "512Mi"
        },
        {
            "name": "DB_VOLUME_CAPACITY",
            "displayName": "Volume Capacity",
            "description": "Volume space available for data, e.g. 512Mi, 2Gi",
            "value": "512Mi",
            "required": true
        },
        {
            "name": "DATABASE_DATA_VOLUME",
            "displayName": "Volumne name for DB data",
            "required": true,
            "value": "mongodb-data"
        },
        {
            "name": "DATABASE_SERVICE_NAME",
            "displayName": "Database Service Name",
            "required": true,
            "value": "mongodb"
        },
        {
            "name": "DATABASE_NAME",
            "displayName": "Database Name",
            "required": true,
            "value": "test1"
        },
        {
            "name": "DATABASE_USER",
            "displayName": "Database Username",
            "required": false    
        },
        {
            "name": "DATABASE_PASSWORD",
            "displayName": "Database User Password",
            "required": false    
        }
    ]
}

Is there any issue with my template ? Is it a OpenShift issue ? Where and how can I get further details about the mount problem in OpenShift logs ?

-- Roberto
docker
docker-volume
kubernetes
mongodb
openshift

1 Answer

11/16/2016

So, I think you're coming up against 2 different issues.

  1. You're template is setup to pull from the Mongo image on Dockerhub (specified by the blank "namespace" value. When trying to pull the mongo:latest image from Dockerhub in the Web UI, you are greeted by a friendly message notifying you that the docker image is not usable because it runs as root: OpenShift error
  2. OpenShift Online Dev preview has been having some issues related to PVC recently (http://status.preview.openshift.com/). Specifically this reported bug at the moment, https://bugzilla.redhat.com/show_bug.cgi?id=1392650. This may be a cause for some issues, as the "official" Mongo image on OpenShift is also failing to build.

I would like to direct you to an OpenShift MongoDB template, not the exact one used in the Developer Preview, but should hopefully provide some good direction going forward! https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_examples/files/examples/v1.4/db-templates/mongodb-persistent-template.json

-- Will Gordon
Source: StackOverflow