spark on k8s - Error 'Invalid initial heap size: -Xms'

8/10/2018

I am trying to use spark on k8s.

Launched minikube

minikube --memory 8192 --cpus 2 start

and build spark master version (fresh fetched) and build docker image and push to docker hub and issued command.

$SPARK_HOME/bin/spark-submit \
  --master k8s://192.168.99.100:8443 \
  --deploy-mode cluster --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=ruseel/spark:testing \ 
  local:///tmp/spark-examples_2.11-2.4.0-SNAPSHOT-shaded.jar

But pod log said

...
+ case "$SPARK_K8S_CMD" in
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_JAVA_OPTS[@]}" -cp "$SPARK_CLASSPATH" -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $SPARK_DRIVER_ARGS)
+ exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -cp ':/opt/spark/jars/*' -Xms -Xmx -Dspark.driver.bindAddress=172.17.0.4
Invalid initial heap size: -Xms
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

How can I run this command?

-- ruseel
apache-spark
kubernetes

1 Answer

8/13/2018

Spark master's new entrypoint.sh is not using $SPARK_DRIVER_MEMORY. It seems to be removed in this commit. So this error doesn't raised anymore for me.

-- ruseel
Source: StackOverflow