I am trying to use spark on k8s.
Launched minikube
minikube --memory 8192 --cpus 2 start
and build spark master version (fresh fetched) and build docker image and push to docker hub
and issued command.
$SPARK_HOME/bin/spark-submit \
--master k8s://192.168.99.100:8443 \
--deploy-mode cluster --name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.container.image=ruseel/spark:testing \
local:///tmp/spark-examples_2.11-2.4.0-SNAPSHOT-shaded.jar
But pod log said
...
+ case "$SPARK_K8S_CMD" in
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_JAVA_OPTS[@]}" -cp "$SPARK_CLASSPATH" -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $SPARK_DRIVER_ARGS)
+ exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -cp ':/opt/spark/jars/*' -Xms -Xmx -Dspark.driver.bindAddress=172.17.0.4
Invalid initial heap size: -Xms
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
How can I run this command?
Spark master's new entrypoint.sh is not using $SPARK_DRIVER_MEMORY
. It seems to be removed in this commit. So this error doesn't raised anymore for me.