I am able to connect to Spark 2.4 on Kubernetes. Here's my code below for the configuration:

conf.setMaster("k8s://https://192.168.1.1:6443") \
    .set("spark.submit.deployMode",'client') \
    .set("spark.kubernetes.namespace","spark-project1") \
    .set("spark.driver.host","192.168.1.1") \
    .set("spark.driver.port","7787") \
    .set("spark.kubernetes.driver.volumes.persistentVolumeClaim.volume.mount.path","/opt/spark/work-dir") \
    .set("spark.kubernetes.driver.volumes.persistentVolumeClaim.volume.mount.readOnly","False") \
    .set("spark.kubernetes.driver.volumes.persistentVolumeClaim.volume.options.claimName","myclaim") \
    .set("spark.kubernetes.executor.volumes.persistentVolumeClaim.volume.mount.path","/opt/spark/work-dir") \
    .set("spark.kubernetes.executor.volumes.persistentVolumeClaim.volume.mount.readOnly","False") \
    .set("spark.kubernetes.executor.volumes.persistentVolumeClaim.volume.options.claimName","myclaim") \
    .set("spark.kubernetes.container.image","192.168.1.1:5000/spark/spark-py:v1") \
    .set("spark.executor.instances","3") \
    .set("spark.kubernetes.pyspark.pythonVersion","3") \
    .set("spark.executor.memory", "100g") \
    .set("spark.kubernetes.authenticate.driver.serviceAccountName","spark")

However I can see container workers being created having only 1 core and a total of 185Gb of memory.

What am I missing here?