Dask Helm Chart - How to create Dask-CUDA-Worker nodes

6/29/2020

I installed the Dask Helm Chart with a revised values.yaml to have 10 workers, however instead of Dask Workers I want to create Dash CUDA Workers to take advantage of the NVIDIA GPUs on my multi-node cluster.

I tried to modify the values.yaml as follows to get Dask CUDA workers instead of Dask Workers, but the worker pods are able to start. I did already install the NVIDIA GPUs on all my nodes on the Kubernetes per the official instructions so I'm not sure what DASK needs to see in order to create the Dask-Cuda-Workers.

worker:
  name: worker
  image:
    repository: "daskdev/dask"
    tag: 2.19.0
    pullPolicy: IfNotPresent
    dask_worker: "dask-cuda-worker"
    #dask_worker: "dask-worker"
    pullSecrets:
    #  - name: regcred
  replicas: 15
  default_resources:  # overwritten by resource limits if they exist
    cpu: 1
    memory: "4GiB"
  env:
  #  - name: EXTRA_APT_PACKAGES
  #    value: build-essential openssl
  #  - name: EXTRA_CONDA_PACKAGES
  #    value: numba xarray -c conda-forge
  #  - name: EXTRA_PIP_PACKAGES
  #    value: s3fs dask-ml --upgrade
  resources: {}
  #  limits:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
  #  requests:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
-- Vinh Tran
dask
kubernetes
worker

1 Answer

6/29/2020

As dask-cuda-worker is not yet officially in the dask image you will need to pull the image a different image: rapidsai/rapidsai:latest

-- quasiben
Source: StackOverflow