I'm trying to deploy a Spark job on Kubernetes, using kubectl apply -f <config_file.yml> (after building Docker image based on Dockerfile). The pod is successfuly created on K8s, then quickly stops with a Failed status. Nothing in the logs help understanding where the error comes from. Other jobs have been successfully deployed on the K8s cluster using the same Dockerfile and config file.

The spark job is supposed to read data from a kafka topic, parse it and outout it in console.

Any idea what might be causing the job to fail?

Dockerfile, built using docker build --rm -f "Dockerfile" xxxxxxxx:80/apache/myapp-test . && docker push xxxxxxxx:80/apache/myapp-test :

FROM xxxxxxxx:80/apache/spark:v2.4.4-gcs-prometheus

#USER root

ADD myapp.jar /jars

RUN adduser --no-create-home --system spark

RUN chown -R spark /prometheus /opt/spark

USER spark

config_file.yml :

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: myapp
  namespace: spark
  labels:
    app: myapp-test
    release: spark-2.4.4
spec:
  type: Java
  mode: cluster
  image: "xxxxxxxx:80/apache/myapp-test"
  imagePullPolicy: Always
  mainClass: spark.jobs.app.streaming.Main
  mainApplicationFile: "local:///jars/myapp.jar"
  sparkVersion: "2.4.4"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 5
    onFailureRetryInterval: 30
    onSubmissionFailureRetries: 0
    onSubmissionFailureRetryInterval: 0
  driver:
    cores: 1
    memory: "1G"
    labels:
      version: 2.4.4
monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    prometheus:
      jmxExporterJar: "/prometheus/jmx_prometheus_javaagent-0.11.0.jar"
      port: 8090
  imagePullSecrets:
  - xxx

Logs :

++ id -u
+ myuid=100
++ id -g
+ mygid=65533
+ set +e
++ getent passwd 100
+ uidentry='spark:x:100:65533:Linux User,,,:/home/spark:/sbin/nologin'
+ set -e
+ '[' -z 'spark:x:100:65533:Linux User,,,:/home/spark:/sbin/nologin' ']'
+ SPARK_K8S_CMD=driver
+ case "$SPARK_K8S_CMD" in
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ + sed sort -t_ 's/[^=]*=\(.*\)/\1/g'-k4 
-n
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ PYSPARK_ARGS=
+ '[' -n '' ']'
+ R_ARGS=
+ '[' -n '' ']'
+ '[' '' == 2 ']'
+ '[' '' == 3 ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=192.168.225.14 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class spark.jobs.app.streaming.Main spark-internal
20/04/20 09:27:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (org.apache.spark.deploy.SparkSubmit$anon$2).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Pod events as shown with kubectl describe pod :

Events:
  Type     Reason            Age                From                             Message
  ----     ------            ----               ----                             -------
  Normal   Scheduled         15m                default-scheduler                Successfully assigned spark/myapp-driver to xxxxxxxx.preprod.local
  Warning  FailedMount       15m                kubelet, xxxxxxxx.preprod.local  MountVolume.SetUp failed for volume "spark-conf-volume" : configmap "myapp-1587388343593-driver-conf-map" not found
  Warning  DNSConfigForming  15m (x4 over 15m)  kubelet, xxxxxxxx.preprod.local  Search Line limits were exceeded, some search paths have been omitted, the applied search line is: spark.svc.cluster.local svc.cluster.local cluster.local preprod.local
  Normal   Pulling           15m                kubelet, xxxxxxxx.preprod.local  Pulling image "xxxxxxxx:80/apache/myapp-test"
  Normal   Pulled            15m                kubelet, xxxxxxxx.preprod.local  Successfully pulled image "xxxxxxxx:80/apache/myapp-test"
  Normal   Created           15m                kubelet, xxxxxxxx.preprod.local  Created container spark-kubernetes-driver
  Normal   Started           15m                kubelet, xxxxxxxx.preprod.local  Started container spark-kubernetes-driver