How to use HDFS HA in spark on k8s?

10/21/2019

My environment is CDH5.11 with HDFS HA mode,I submit application use SparkLauncher in my windows PC,when I code like

setAppResource("hdfs://ip:port/insertInToSolr.jar")

it worked,but when I code like

setAppResource("hdfs://hdfsHA_nameservices/insertInToSolr.jar")

it does not work. I have copy my hadoop config to the spark docker image by modify

$SPARK_HOME/kubernetes/dockerfiles/spark/entrypoint.sh When I use docker run -it IMAGE ID /bin/bash to run a CONTAINER, in the CONTAINER I can use spark-shell to read hdfs and hive.

-- tzxxh
apache-spark
kubernetes

0 Answers