Is it possible to run spark in client mode with driver on host system instead of a separate pod in Kubernetes

5/17/2020

In spark documentation, we have info about how to run spark driver in a separate pod in Client Mode Networking. But we don't have any information about running the driver in host system itself instead of running the driver in separate pod in a kubernetes cluster. Have the following questions.

  • How to we explicitly configure to run driver in a separate pod or
    running in the host system itself. What is the benefit of having
    driver run in separate pod

I am facing issues with running in the host system (guess this is the default mode when client mode is mentioned when running in k8s and only executor images are provided in the config.) . Driver not able access executor with ip address to fetch block files. I can see that its because the driver and executors are in two different networks. enter image description here

I See many blogs to run in client mode with spark driver in a separate pod, but not for running it in the host system itself.

Any suggestions or directions will be greatly helpful

-- user1833930
apache-spark
kubernetes

0 Answers