I downloaded spark binaries, I created a docker image of dockerfile present under /kubernetes/dockerfiles and followed this documentation - https://spark.apache.org/docs/latest/running-on-kubernetes.html
I am trying to submit a spark job as follows -
~/spark-2.3.0-bin-hadoop2.7/bin/spark-submit --master
k8s://http://<k8s-api-server>:<k8s-port> --deploy-mode cluster --
name spark-pi --class org.apache.spark.examples.SparkPi --
conf spark.executor.instances=5 --conf
spark.kubernetes.container.image=spark:1.0 --conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar
I am getting the following error -
++ id -u
+ myuid=0
++ id -g
+ mygid=0
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=driver
+ '[' -z driver ']'
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_JAVA_OPTS
+ '[' -n /opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar ']'
+ SPARK_CLASSPATH=':/opt/spark/jars/*:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar'
+ '[' -n '' ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_JAVA_OPTS[@]}" -cp "$SPARK_CLASSPATH" -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $SPARK_DRIVER_ARGS)
+ exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -Dspark.driver.host=spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver-svc.default.svc -Dspark.kubernetes.container.image=spark:1.0 -Dspark.driver.port=7078 -Dspark.kubernetes.executor.podNamePrefix=spark-pi-560b11363c2f360b8d13f1d56eb3966c -Dspark.driver.blockManager.port=7079 -Dspark.kubernetes.driver.pod.name=spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver -Dspark.kubernetes.authenticate.driver.serviceAccountName=spark -Dspark.executor.instances=5 -Dspark.app.name=spark-pi -Dspark.master=k8s://http://10.118.202.130:8080 -Dspark.submit.deployMode=cluster -Dspark.jars=/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar,/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar -Dspark.app.id=spark-810f3f33b27844da9f584bb4ef661f45 -cp ':/opt/spark/jars/*:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar' -Xms1g -Xmx1g -Dspark.driver.bindAddress=10.118.202.118 org.apache.spark.examples.SparkPi
2018-05-21 20:59:30 INFO SparkContext:54 - Running Spark version 2.3.0
2018-05-21 20:59:31 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-21 20:59:31 INFO SparkContext:54 - Submitted application: Spark Pi
2018-05-21 20:59:31 INFO SecurityManager:54 - Changing view acls to: root
2018-05-21 20:59:31 INFO SecurityManager:54 - Changing modify acls to: root
2018-05-21 20:59:31 INFO SecurityManager:54 - Changing view acls groups to:
2018-05-21 20:59:31 INFO SecurityManager:54 - Changing modify acls groups to:
2018-05-21 20:59:31 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2018-05-21 20:59:31 INFO Utils:54 - Successfully started service 'sparkDriver' on port 7078.
2018-05-21 20:59:31 INFO SparkEnv:54 - Registering MapOutputTracker
2018-05-21 20:59:31 INFO SparkEnv:54 - Registering BlockManagerMaster
2018-05-21 20:59:31 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-05-21 20:59:31 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-05-21 20:59:31 INFO DiskBlockManager:54 - Created local directory at /tmp/blockmgr-10dc281e-86b2-4f82-ae46-531ee0db01ee
2018-05-21 20:59:31 INFO MemoryStore:54 - MemoryStore started with capacity 408.9 MB
2018-05-21 20:59:31 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2018-05-21 20:59:31 INFO log:192 - Logging initialized @1651ms
2018-05-21 20:59:31 INFO Server:346 - jetty-9.3.z-SNAPSHOT
2018-05-21 20:59:32 INFO Server:414 - Started @1729ms
2018-05-21 20:59:32 INFO AbstractConnector:278 - Started ServerConnector@66629f63{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-05-21 20:59:32 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@493dfb8e{/jobs,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@15a902e7{/jobs/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7876d598{/jobs/job,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5af28b27{/jobs/job/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@71104a4{/stages,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4985cbcb{/stages/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@72f46e16{/stages/stage,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@549621f3{/stages/stage/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54361a9{/stages/pool,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@32232e55{/stages/pool/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5217f3d0{/storage,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@37ebc9d8{/storage/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@293bb8a5{/storage/rdd,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2416a51{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6fa590ba{/environment,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e9319f{/environment/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@72e34f77{/executors,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7bf9b098{/executors/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@389adf1d{/executors/threadDump,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77307458{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1fc0053e{/static,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ec0c838{/,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e46d9f4{/api,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7997b197{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@11dee337{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-05-21 20:59:32 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver-svc.default.svc:4040
2018-05-21 20:59:32 INFO SparkContext:54 - Added JAR /opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar at spark://spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver-svc.default.svc:7078/jars/spark-examples_2.11-2.3.0.jar with timestamp 1526936372118
2018-05-21 20:59:32 WARN KubernetesClusterManager:66 - The executor's init-container config map is not specified. Executors will therefore not attempt to fetch remote or submitted dependencies.
2018-05-21 20:59:32 WARN KubernetesClusterManager:66 - The executor's init-container config map key is not specified. Executors will therefore not attempt to fetch remote or submitted dependencies.
2018-05-21 20:59:42 ERROR SparkContext:91 - Error initializing SparkContext.
org.apache.spark.SparkException: External scheduler cannot be instantiated
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$createTaskScheduler(SparkContext.scala:2747)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver] in namespace: [default] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:228)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:184)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.<init>(KubernetesClusterSchedulerBackend.scala:70)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:120)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$createTaskScheduler(SparkContext.scala:2741)
... 8 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:281)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:312)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:295)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:783)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:217)
... 12 more
2018-05-21 20:59:42 INFO AbstractConnector:318 - Stopped
Spark@66629f63{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-05-21 20:59:42 INFO SparkUI:54 - Stopped Spark web UI at http://spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver-svc.default.svc:4040
2018-05-21 20:59:42 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-05-21 20:59:42 INFO MemoryStore:54 - MemoryStore cleared
2018-05-21 20:59:42 INFO BlockManager:54 - BlockManager stopped
2018-05-21 20:59:42 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2018-05-21 20:59:42 WARN MetricsSystem:66 - Stopping a MetricsSystem that is not running
2018-05-21 20:59:42 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-05-21 20:59:42 INFO SparkContext:54 - Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: External scheduler cannot be instantiated
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$createTaskScheduler(SparkContext.scala:2747)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:492)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:31)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [spark-pi-560b11363c2f360b8d13f1d56eb3966c-driver] in namespace: [default] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:228)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:184)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend.<init>(KubernetesClusterSchedulerBackend.scala:70)
at org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager.createSchedulerBackend(KubernetesClusterManager.scala:120)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$createTaskScheduler(SparkContext.scala:2741)
... 8 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:281)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151)
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195)
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
at okhttp3.RealCall.execute(RealCall.java:69)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:377)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:312)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:295)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:783)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:217)
... 12 more
2018-05-21 20:59:42 INFO ShutdownHookManager:54 - Shutdown hook called
2018-05-21 20:59:42 INFO ShutdownHookManager:54 - Deleting directory
/tmp/spark-286e2bfe-9a7a-490a-aade-88fd81ff49c4
Please help.
Thanks.
Im just guessing here, is your kube cluster running with rbac enabled? If yes, did you apply the rbac ?
https://spark.apache.org/docs/latest/running-on-kubernetes.html#rbac