I am using k8s to deploy spark invoked using spark-submit. We use contour as our ingress-class. I was wondering if there is way to create ingress object for the spark-driver container to expose Spark UI. I am trying to see if all this can be done in one step process may be by using annotations or labels. Something like hxquangnhat/kubernetes-auto-ingress, which uses annotation to enable ingress
All I want to do is to use spark-submit to submit the spark job and get the Saprk UI exposed. May be create the ingress using --conf like
--conf spark.kubernetes.driver.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.executor.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/port=XXXXX \
--conf spark.kubernetes.executor.annotation.prometheus.io/port=XXXXX \Please let me know if you have any thoughts or have seen some examples like this.
you can simply create Ingress right after Spark Driver submission. Simply provide a script with a language of your choice having Kubernetes client library.
In order to configure automatic Ingress deletion/GC on Spark Driver Pod deletion you can use Kubernetes OwnerReference to Spark Driver Pod.
Also you may want to refer the Apache Livy project Spark on Kubernetes support PR and the related Helm charts repo, which offer a way to solve Spark UI exposure as well as some other aspects of running Spark on Kubernetes.