accessing error while submitting spark job from local to spark cluster which is located on server in eclipse

11/20/2019

I am trying to run a simple spark program in my eclipse ide, and i am using spark master url but i am getting a error again and again. Please find the error which i am facing

2019-11-20 11:49:58 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 9 from BlockManagerMaster.
2019-11-20 11:49:58 INFO  BlockManagerMaster:54 - Removal of executor 9 requested
2019-11-20 11:49:58 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 9
2019-11-20 11:50:01 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:06 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:07 INFO  BlockManagerMaster:54 - Removal of executor 10 requested
2019-11-20 11:50:07 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 10
2019-11-20 11:50:07 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 10 from BlockManagerMaster.
2019-11-20 11:50:11 WARN  TaskSchedulerImpl:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-11-20 11:50:11 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:13 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:17 INFO  BlockManagerMaster:54 - Removal of executor 11 requested
2019-11-20 11:50:17 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 11
2019-11-20 11:50:17 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 11 from BlockManagerMaster.
2019-11-20 11:50:19 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:23 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:25 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 12 from BlockManagerMaster.
2019-11-20 11:50:25 INFO  BlockManagerMaster:54 - Removal of executor 12 requested
2019-11-20 11:50:25 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 12
2019-11-20 11:50:26 WARN  TaskSchedulerImpl:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-11-20 11:50:28 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:32 INFO  ExecutorPodsAllocator:54 - Going to request 1 executors from Kubernetes.
2019-11-20 11:50:34 INFO  BlockManagerMaster:54 - Removal of executor 13 requested
2019-11-20 11:50:34 INFO  BlockManagerMasterEndpoint:54 - Trying to remove executor 13 from BlockManagerMaster.
2019-11-20 11:50:34 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asked to remove non-existent executor 13"
``
Please find the my spark conf:
 val conf = new SparkConf().setAppName("sparksetuptest").setMaster("k8s://http://10.90.xx.xx:xx").set("spark.cores.max", "1").set("spark.driver.host", "10.95.xx.xx").set("spark.dynamicAllocation.enabled","false").set("spark.shuffle.service.enabled","false")
      .set("spark.driver.allowMultipleContexts", "true")
      .set("spark.io.compression.codec", "snappy")
      .set("spark.rdd.compress", "true")
      .set("spark.local.ip","10.90.xx.xx")
      .set("spark.kubernetes.container.image", "shreet007/spark")
      .set("spark.submit.deployMode","cluster")
      .set("spark.kubernetes.executor.limit.cores","2")
      .set("spark.executor.instances", "3")
      .set("spark.executor.memory","2g")
      .set("spark.dynamicAllocation.minExecutors", "1")
      .set("spark.dynamicAllocation.maxExecutors", "3")
-- harshit saxena
apache-spark
eclipse
kubernetes
scala

0 Answers