Host O/S Version: CentOS 7.3

Docker Version:1.12.6

CUDA Version 8.0.61

There are 4 GPUs on the host machine. Below are the details:

"Devices": [
                {
                    "PathOnHost": "/dev/nvidiactl",
                    "PathInContainer": "/dev/nvidiactl",
                    "CgroupPermissions": "mrw"
                },
                {
                    "PathOnHost": "/dev/nvidia-uvm",
                    "PathInContainer": "/dev/nvidia-uvm",
                    "CgroupPermissions": "mrw"
                },
                {
                    "PathOnHost": "/dev/nvidia2",
                    "PathInContainer": "/dev/nvidia0",
                    "CgroupPermissions": "mrw"
                },
                {
                    "PathOnHost": "/dev/fuse",
                    "PathInContainer": "/dev/fuse",
                    "CgroupPermissions": "mrw"
                }
            ],

See below the list of Nvidia devices:

root@de-18--dev-105631-3244950001-48pjz:/sys/fs/cgroup/devices# ls /dev | grep nv
    nvidia-uvm
    nvidia-uvm-tools
    nvidia0
    nvidia1
    nvidia2
    nvidia3
    nvidiactl
root@de-18--dev-105631-3244950001-48pjz:/sys/fs/cgroup/devices# cat devices.list 
    a *:* rwm

Container is not running on the privileged mode. When the number of GPU is correct in the container, but sometime later there we can see 4 GPUs in then container.