I'm trying to set up a new Kubeflow deployment to use preemptible VMs. Referring to this doc, I see that there is some YAML to specify preemptible nodes but I am not sure where to put it in the myriad of YAML files generated by kfctl.sh. Meanwhile, going through the console, I see Preemptible nodes = disabled, but there is no way to change it.
How do I set the Kubeflow node pools to be preemptible?
There's no way to change a node-pool from non-preemptible to preemptible.
What you can do is create a new preemptible node-pool and migrate the workloads from the non-preemptible node-pool to the new one.
By the other hand, you can go ahead and recreate the GKE cluster with a preemprible node-pool.
Hope this helps!