java.lang.ClassNotFoundException: org.apache.spark.deploy.kubernetes.submit.Client

8/22/2017

I am running a sample spark job in kubernetes cluster with following command:

bin/spark-submit \
  --deploy-mode cluster \
  --class org.apache.spark.examples.SparkPi \
  --master k8s://https://XXXXX \
  --kubernetes-namespace sidartha-spark-cluster \
  --conf spark.executor.instances=2 \
  --conf spark.app.name=spark-pi \
  --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.1.0-kubernetes-0.1.0-rc1 \
  --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.1.0-kubernetes-0.1.0-rc1 \
  examples/jars/spark-examples_2.11-2.1.0-k8s-0.1.0-SNAPSHOT.jar 1000

I am building the spark from apache-spark-on-k8s

I am not able find the jar for org.apache.spark.deploy.kubernetes.submit.Client Class.

-- Mike
apache-spark
kubernetes

1 Answer

8/23/2017

This issue is resolved. We need to build the spark/resource-manager/kubernetes from the source.

-- Mike
Source: StackOverflow