We are launching all spark jobs as kubernetics(k8es) containers inside a k8es cluster. We also create a service on each job and do port forwarding for the spark UI (container's 4040 is mapped to SvcPort, say, 31123). The same set of nodes is also hosting a Yarn cluster. Container's entry command is a call to spark-submit to Yarn in client mode.

Now, the spark container may get launched on any of the name-nodes - master or standby. There is a VIP assigned to the active name-node.

When spark container gets launched on active name-node, spark UI and all its tabs are easily accessible from anywhere using VIP:SvcPort However, when spark container gets launched on standby name node, spark UI is NOT accessible and eventually request fails with 500 error (as the VIP:SvcPort gets redirected to ActiveNamenode:8080 which is not accessible as driver is now running on standby name node.

I have tried to reason and try out multiple service configurations but nothing seems to work this scenario. This config is what I have tried using which UI is accessible when container is on master node but UI is NOT accessible when container is on standby node.

apiVersion: v1
kind: Service
metadata:
   name: spark-svc
   labels:
     deployer: sample_app
     app: spark-app
spec:
  selector:
    app: spark-app
  ports:
  - port: 31123
    protocol: TCP
    targetPort: 4040
    name: ui
  externalIPs:
  - 192.168.122.22

May anyone pls suggest proper k8es service config for this scenario. We do not want to restrict the spark containers to master name-node as let k8es address the job load within k8es cluster.

Any help is much appreciated.