What is the proper way to deploy an adaptive multi-user Dask cluster on Kubernetes?
I need a centralized cluster of machines multiple people can use for their work, so that it can add more machines or remove them (preferably, to 0 workers).
Given that you have a requirement for multiple people to share a cluster of machines that you manage I recommend you check out Dask Gateway.
A Dask cluster is only designed to be used by one person at a time. You can create clusters using the Dask Helm Chart or with dask-kubernetes, but each of these methods creates a single cluster for use by one person.
Dask Gateway provides a secure, multi-tenant server for managing Dask clusters.
To get started on Kubernetes you need to create a Helm configuration file (config.yaml) with a gateway proxy token.
gateway:
proxyToken: "<RANDOM TOKEN>"Hint: You can generate a suitable token with openssl rand -hex 32.
Then install the chart.
helm repo add dask-gateway https://dask.org/dask-gateway-helm-repo/
helm repo update
helm install --values config.yaml my-release dask-gateway/dask-gateway