I m trying to run Spark on Kubernetes as Scheduler.

It works fine when running from outside of kubernetes cluster using kubectl proxy.

spark-shell --master k8s://http://localhost:8001 --conf spark.kubernetes.container.image=abdoumediaoptimise/spark

But whenever we try running spark-shell or spark-submit from within a pod directly , it never works (even by following rbac from spark docs with : --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark . We have authorization execution exception:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes/api/v1/namespaces/default/pods?labelSelector=spark-app-selector%3Dspark-application-1574714537374,spark-role%3Dexecutor. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "default"

Any idea how to Launch Spark from within pods ? this actually makes using spark k8s:// with notebooks impossible

Spark RBAC YAML file

apiVersion: v1
kind: ServiceAccount
metadata:
  name:  spark
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: spark
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
  - kind: ServiceAccount
    name: spark
    namespace: default