I recently hosted Jupyterhub in our K8s cluster. We set memory limits to 1GB so now all Jupyter notebook are with this configuration. But we have notebooks which are resource(memory and cpu) intensive and some notebooks are not resource intensive
So , is there anyway to assign different resource for multiple jupyter pods ?
I am using helm chart to deploy jupyterhub https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html
There is no other option if your PODs are managed by Deployment or Stateful set.
Either create the Different different deployments for multiple Jupiter notebooks having different configurations or deploy each notebook as a different POD with resource requirements.
i would suggest using the VPA which will auto-scale your deployment or stateful set and adjust the resource requirement as per need.
Check more about the VPA : https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler
We use profiles to allow users to select from pods of different sizes. The pods are all deployed on large nodes, but users can select whether they get a half- or full-node pod.
As an example, we build off the daskhub configuration (full spec here):
daskhub:
jupyterhub:
singleuser:
image:
cpu:
limit: 3.5
guarantee: 3.5
memory:
limit: 22.5G
guarantee: 22.5G
profileList:
- display_name: "default"
description: "Default notebook size"
default: true
- display_name: "large"
description: "Larger notebook allowance"
kubespawner_override:
cpu_limit: 7.0
cpu_guarantee: 7.0
mem_limit: 45G
mem_guarantee: 45GThese are both spawned on 8CPU / 52GB nodes, but two of the smaller pods will go on the same node (assuming 2 users selecting "default").