I am trying to run spark-on-k8s, While I reading stuff on internet I stumbled upon words 'Resource Staging Server'.
AFAIK that is server to sharing jars to executor. But when I issued commands with spark-submit.
spark-2.3.1-bin-hadoop2.7/bin/spark-submit \
--master k8s://k8s-url \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.container.image=$I \
local:///Users/myid/examples/target/spark-examples_2.11-2.4.0-SNAPSHOT-shaded.jar
That pod seems not to be created.
So where is source of Resource Staging Server(RSS)? Or is RSS still actively developed?
RSS has been removed by Spark since version 2.3, but it still works at 2.2, spark on K8S.
BTW, RSS pod should be created manually before spark-submit.