I'm currently using Docker 19.03 and Kubernetes 1.13.5 and Rancher 2.2.4. Since 19.03, Docker has officially support natively NVIDIA GPUs just by passing --gpus option. Example (from NVIDIA/nvidia-docker github):
docker run --gpus all nvidia/cuda nvidia-smiBut in Kubernetes, there's no option to pass Docker CLI options. So if I need to run a GPU instance, I have to install nvidia-docker2, which is not convenient to use.
Is there anyway to pass the Docker CLI options or passing NVIDIA runtime without installing nvidia-docker2
GPU's are scheduled via device plugins in Kubernetes.
The official NVIDIA GPU device plugin has the following requirements:
- Kubernetes nodes have to be pre-installed with NVIDIA drivers.
- Kubernetes nodes have to be pre-installed with nvidia-docker 2.0
- nvidia-container-runtime must be configured as the default runtime for docker instead of runc.
- NVIDIA drivers ~= 361.93
Once the nodes are setup GPU's become another resource in your spec like cpu or memory.
spec:
containers:
- name: gpu-thing
image: whatever
resources:
limits:
nvidia.com/gpu: 1