K
Q

Run Spark example on Kubernetes failed

5/1/2020
  • My Mac OS/X Version : 10.15.3
  • Minikube Version: 1.9.2

I start the minikube use the following command without any extra configuration.

minikube start --driver=virtualbox
--image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' --cpus 4 --memory 4096 --alsologtostderr

And I download spark-2.4.5-bin-hadoop2.7 from the Spark official website and build spark images by the following command

eval $(minikube docker-env)
./bin/docker-image-tool.sh -m -t 2.4.5 build

then I run Spark-pi using the follwing command within my local machine where store the Spark 2.4.5.

kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=admin --serviceaccount=default:spark --namespace=default
./bin/spark-submit \
--master k8s://https://192.168.99.104:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.container.image=spark:2.4.5 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar

I get the following error

enter image description here

the full log can be found at full log

Can anyone explain this error and how to solve it?

-- Huzhenyu
apache-spark
kubernetes
minikube

Similar Questions

arranging order of fields in logs from ELK
Kubernetes: access from outside
Pod security policy Nonrootuser is not allowing tcpdump inside container
Java RAM Percentage options ignored

2 Answers

5/29/2020

Another spark image (from gcr.io/spark-operator/spark) worked for me, without downgrading the version of Kubernetes.

bin/spark-submit \
--master k8s://https://192.168.99.100:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=512m \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=512m \
--conf spark.executor.cores=1 \
--conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.5 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar
-- Kien Mai
Source: StackOverflow
5/3/2020

Please check the Kubernetes version you launched with Minikube.

Spark v2.4.5 fabric8 Kubernetes client v4.6.1 is compatible with Kubernetes API up to 1.15.2 (refer answer).

You can launch the specific Kubernetes API version with Minikube by adding --kubernetes-version flag to minikube start command (refer docs).

Also the issue might be caused by OkHttp library bug described in the comment of this qustion.

-- Aliaksandr Sasnouskikh
Source: StackOverflow