In spark documentation, we have info about how to run spark driver in a separate pod in Client Mode Networking. But we don't have any information about running the driver in host system itself instead of running the driver in separate pod in a kubernetes cluster. Have the following questions.
I am facing issues with running in the host system (guess this is the default mode when client mode is mentioned when running in k8s and only executor images are provided in the config.) . Driver not able access executor with ip address to fetch block files. I can see that its because the driver and executors are in two different networks.
I See many blogs to run in client mode with spark driver in a separate pod, but not for running it in the host system itself.
Any suggestions or directions will be greatly helpful