I am trying to use spark on Kubernetes cluster (existing setup: emr + yarn). Our use case is to handle too many jobs including short lived ones (few seconds to 15 minutes). Also, we have peak hours when many workers needs to run to handle 100s of jobs running concurrently.

So what I want to achieve, running master and fixed few workers (say 5) all time and increase workers to 40-50 at peak time. Also, I will prefer to use dynamic allocation.

I am setting it as below

Master image (spark-master:X)

	FROM <BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark>
	ENTRYPOINT ["/opt/spark/sbin/start-master.sh", "-p", "8081", "<A long running server command that can accept get traffic on 8080 to submit jobs>"]

Worker worker image (spark-worker:X)

	FROM <BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark>
	ENTRYPOINT ["/opt/spark/sbin/start-worker.sh", "spark//spark-master:8081" ,"-p", "8081", "<A long running server command to keep up the worker>"]

Deplyments

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-master-server
spec:
  replicas: 1
  selector:
    matchLabels:
      component: spark-master-server
  template:
    metadata:
      labels:
        component: spark-master-server
    spec:
      containers:
        - name:  spark-master-server
          image: spark-master:X
          imagePullPolicy: IfNotPresent 
          ports:
            - containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: spark-master

spec:
  type: ClusterIP
  ports:
    - port: 8081
      targetPort: 8081
  selector:
    component: spark-master-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-worker-instance
spec:
  replicas: 3
  selector:
    matchLabels:
      component: spark-worker-instance
  template:
    metadata:
      labels:
        component: spark-worker-instance
    spec:
      containers:
        - name:  spark-worker-server
          image: spark-worker:X
          imagePullPolicy: IfNotPresent 
          ports:
            - containerPort: 8081

Questions

• Is this setup recommended ?
• How can we submit new job from within Kubernetes cluster in absence of yarn (k8s) ?

Reason we are trying not create master and driver dynamically per job (as given in example - http://spark.apache.org/docs/latest/running-on-kubernetes.html) is that it may be an overhead for large no. of small jobs.