would like to know if anyone tried Oozie with spark 2.3 on kubernetes
can i just place spark 2.3 under sharedlib and use spark action with k8smaster as master in workflow xml?
Oozie 5.0.0 works with Spark 2.3 on Kubenetes, you need to add Spark-Kubernetes jar to sharelib by adding
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-kubernetes_2.11</artifactId>
<version>2.3.0</version>
<scope>compile</scope>
</dependency>
to sharelib/spark/pom.xml
and build oozie distro with Spark 2.3 support using command
./bin/mkdistro.sh -DskipTests -Pspark-2 -Dspark.version=2.3.0
-Dspark.scala.binary.version=2.11 -Djackson.version=2.6.5
then configure Spark action with Kubernetes URL:
<spark xmlns="uri:oozie:spark-action:1.0">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<master>k8s://https://hostname</master>
<mode>cluster</mode>
<name>jobname</name>
<jar>jarpath</jar>
<spark-opts>sparkopts</spark-opts>
</spark>