I am trying to run jupyter for spark on k8s.
After little experiment :) I found that
k8s does not support client mode
$ bin/spark-shell --master k8s://192.168.99.100:8443
...
...
Error: Client mode is currently not supported for Kubernetes.And that
spark-shell can't run in cluster mode
$ bin/spark-shell --master k8s://192.168.99.100:8443 --deploy-mode cluster
...
...
Error: Cluster deploy mode is not applicable to Spark shells.So I am guessing that jupyter spark kernel or zeppelin will not run for k8s. (not sure)
As a workaround I am thinking about using livy.
Does apache livy 0.5.0 supports spark on k8s? Any advice for setting up spark cluster would be appreciated.
P.S. I tried to use GCP dataproc but after lost GCS shared note once (during 7 days period evaluation, no clue about reason). I am sure it's my fault (maybe on configuration?) but didn't dive deep to dive deep in k8s. :)
You can try customized livy-0.6.0 for that: https://github.com/apache/incubator-livy/pull/167 We've required a solution for the project to run Spark on Kubernetes and modified Livy to serve that purpose. You can be interested in Helm chart for Livy setup either: https://github.com/jahstreet/spark-on-kubernetes-helm