Where is spark k8s resource staging server (RSS)?

8/14/2018

I am trying to run spark-on-k8s, While I reading stuff on internet I stumbled upon words 'Resource Staging Server'.

AFAIK that is server to sharing jars to executor. But when I issued commands with spark-submit.

spark-2.3.1-bin-hadoop2.7/bin/spark-submit \
  --master k8s://k8s-url \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=$I \
  local:///Users/myid/examples/target/spark-examples_2.11-2.4.0-SNAPSHOT-shaded.jar

That pod seems not to be created.

So where is source of Resource Staging Server(RSS)? Or is RSS still actively developed?

-- ruseel
apache-spark
kubernetes

1 Answer

6/13/2019

RSS has been removed by Spark since version 2.3, but it still works at 2.2, spark on K8S.

BTW, RSS pod should be created manually before spark-submit.

-- runzhliu
Source: StackOverflow