How can I run spark-submit commands using the GCP spark operator on kubernetes

5/7/2020

I have a spark application which i want to deploy on kubernetes using the GCP spark operatorhttps://github.com/GoogleCloudPlatform/spark-on-k8s-operator.
I was able to run a spark application using command kubectl apply -f example.yaml but i want to use spark-submit commands.

-- Dhruv Singh Chandel
apache-spark
google-cloud-platform
google-kubernetes-engine
kubernetes
spark-submit

1 Answer

5/7/2020

There are few options mentione by https://github.com/big-data-europe/docker-spark which can use see if that solves your problem

kubectl run spark-base --rm -it --labels="app=spark-client" --image bde2020/spark-base:2.4.5-hadoop2.7 -- bash ./spark/bin/spark-shell --master spark://spark-master:7077 --conf spark.driver.host=spark-client

or

kubectl run spark-base --rm -it --labels="app=spark-client" --image bde2020/spark-base:2.4.5-hadoop2.7 -- bash ./spark/bin/spark-submit --class CLASS_TO_RUN --master spark://spark-master:7077 --deploy-mode client --conf spark.driver.host=spark-client URL_TO_YOUR_APP
-- QuickSilver
Source: StackOverflow