I am attempting to deploy an adaptive dask kubernetes cluster to my aws K8s instance (I want to use the kubeControl interface found here). It is unclear to me where and how I execute this code such that it is active on my existing cluster. In addition to this, I want to have an ingress rule such that another ec2 instance I have can connect to the cluster and execute code within an aws VPC to maintain security and network performance.

So far I have managed to get a functional k8s cluster running with dask and jupyterhub running on it. I am using the sample helm chart found here which reference the docker image here. I can see this image does not even install dask-kubernetes. With that being said, I am able to connect to this cluster from my other ec2 instance using the exposed AWS dns server and execute custom code but this is not the kubernetes native dask cluster.

I have worked on modifying the deploy yaml for kubernetes but it is unclear to me what I would need to change to have it use the proper kubernetes cluster/schedulers. I do know I need to modify the docker image I am using to have in install dask-kubernetes, but this still does not help me. Below is the sample helm deploy chart I am using

---
# nameOverride: dask
# fullnameOverride: dask

scheduler:
  name: scheduler
  image:
    repository: "daskdev/dask"
    tag: 2.3.0
    pullPolicy: IfNotPresent
    # See https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    pullSecrets:
    #  - name: regcred
  replicas: 1
  # serviceType: "ClusterIP"
  # serviceType: "NodePort"
  serviceType: "LoadBalancer"
  servicePort: 8786
  resources: {}
  #  limits:
  #    cpu: 1.8
  #    memory: 6G
  #  requests:
  #    cpu: 1.8
  #    memory: 6G
  tolerations: []
  nodeSelector: {}
  affinity: {}

webUI:
  name: webui
  servicePort: 80

worker:
  name: worker
  image:
    repository: "daskdev/dask"
    tag: 2.3.0
    pullPolicy: IfNotPresent
    # dask_worker: "dask-cuda-worker"
    dask_worker: "dask-worker"
    pullSecrets:
    #  - name: regcred
  replicas: 3
  aptPackages: >-
  default_resources:  # overwritten by resource limits if they exist
    cpu: 1
    memory: "4GiB"
  env:
  #  - name: EXTRA_CONDA_PACKAGES
  #    value: numba xarray -c conda-forge
  #  - name: EXTRA_PIP_PACKAGES
  #    value: s3fs dask-ml --upgrade
  resources: {}
  #  limits:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
  #  requests:
  #    cpu: 1
  #    memory: 3G
  #    nvidia.com/gpu: 1
  tolerations: []
  nodeSelector: {}
  affinity: {}

jupyter:
  name: jupyter
  enabled: true
  image:
    repository: "daskdev/dask-notebook"
    tag: 2.3.0
    pullPolicy: IfNotPresent
    pullSecrets:
    #  - name: regcred
  replicas: 1
  # serviceType: "ClusterIP"
  # serviceType: "NodePort"
  serviceType: "LoadBalancer"
  servicePort: 80
  # This hash corresponds to the password 'dask'
  password: 'sha1:aae8550c0a44:9507d45e087d5ee481a5ce9f4f16f37a0867318c'
  env:
  #  - name: EXTRA_CONDA_PACKAGES
  #    value: "numba xarray -c conda-forge"
  #  - name: EXTRA_PIP_PACKAGES
  #    value: "s3fs dask-ml --upgrade"
  resources: {}
  #  limits:
  #    cpu: 2
  #    memory: 6G
  #  requests:
  #    cpu: 2
  #    memory: 6G
  tolerations: []
  nodeSelector: {}
  affinity: {}