Ozzie with Spark 2.3 on kubernetes

5/13/2018

would like to know if anyone tried Oozie with spark 2.3 on kubernetes

can i just place spark 2.3 under sharedlib and use spark action with k8smaster as master in workflow xml?

-- shiv455
kubernetes
oozie
oozie-coordinator

1 Answer

5/31/2018

Oozie 5.0.0 works with Spark 2.3 on Kubenetes, you need to add Spark-Kubernetes jar to sharelib by adding

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-kubernetes_2.11</artifactId>
      <version>2.3.0</version>
      <scope>compile</scope>
    </dependency>

to sharelib/spark/pom.xml

and build oozie distro with Spark 2.3 support using command

./bin/mkdistro.sh -DskipTests -Pspark-2 -Dspark.version=2.3.0
-Dspark.scala.binary.version=2.11 -Djackson.version=2.6.5

then configure Spark action with Kubernetes URL:

    <spark xmlns="uri:oozie:spark-action:1.0">
        <job-tracker>${resourceManager}</job-tracker>
        <name-node>${nameNode}</name-node>
        <master>k8s://https://hostname</master>
        <mode>cluster</mode>
        <name>jobname</name>
        <jar>jarpath</jar>
        <spark-opts>sparkopts</spark-opts>
    </spark>
-- Igor Shikin
Source: StackOverflow