K
Q

How to use local Docker images when submitting Spark jobs (2.3) natively to Kubernetes?

3/15/2018

I am trying to submit a Spark Job on Kubernetes natively using Apache Spark 2.3. When I use a Docker image on Docker Hub (for Spark 2.2), it works:

bin/spark-submit \
    --master k8s://http://localhost:8080 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
    local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar

However, when I try to build a local Docker image,

sudo docker build -t spark:2.3 -f kubernetes/dockerfiles/spark/Dockerfile .

and submit the job as:

bin/spark-submit \
--master k8s://http://localhost:8080 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--conf spark.kubernetes.container.image=spark:2.3 \
local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar

I get the following error; that is "repository docker.io/spark not found: does not exist or no pull access, reason=ErrImagePull, additionalProperties={})"

status: [ContainerStatus(containerID=null, image=spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = 2 desc = repository docker.io/spark not found: does not exist or no pull access, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]
2018-03-15 11:09:54 INFO  LoggingPodStatusWatcherImpl:54 - State changed, new state: 
     pod name: spark-pi-3a1a6e8ce615395fa7df81eac06d58ed-driver
     namespace: default
     labels: spark-app-selector -> spark-8d9fdaba274a4eb69e28e2a242fe86ca, spark-role -> driver
     pod uid: 5271602b-2841-11e8-a78e-fa163ed09d5f
     creation time: 2018-03-15T11:09:25Z
     service account name: default
     volumes: default-token-v4vhk
     node name: mlaas-p4k3djw4nsca-minion-1
     start time: 2018-03-15T11:09:25Z
     container images: spark:2.3
     phase: Pending
     status: [ContainerStatus(containerID=null, image=spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=Back-off pulling image "spark:2.3", reason=ImagePullBackOff, additionalProperties={}), additionalProperties={}), additionalProperties={})]

Also, I tried to run a local Docker registry as described in: https://docs.docker.com/registry/deploying/#run-a-local-registry

docker run -d -p 5000:5000 --restart=always --name registry registry:2

sudo docker tag spark:2.3 localhost:5000/spark:2.3

sudo docker push localhost:5000/spark:2.3

I can do this successfully: docker pull localhost:5000/spark:2.3

However, when I submit the Spark job:

bin/spark-submit \
    --master k8s://http://localhost:8080 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=localhost:5000/spark:2.3 \
    local:///home/fedora/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar

I again got ErrImagePull:

status: [ContainerStatus(containerID=null, image=localhost:5000/spark:2.3, imageID=, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=false, restartCount=0, state=ContainerState(running=null, terminated=null, waiting=ContainerStateWaiting(message=rpc error: code = 2 desc = Error while pulling image: Get http://localhost:5000/v1/repositories/spark/images: dial tcp [::1]:5000: getsockopt: connection refused, reason=ErrImagePull, additionalProperties={}), additionalProperties={}), additionalProperties={})]

Is there a way in Spark 2.3 to use local Docker images when submitting jobs natively to Kubernetes?

Thank you in advance.

-- Canan
apache-spark
docker
kubernetes

Similar Questions

FluentD fails flוshing to ElasticSearch (K8S, AWS)
Impossible to use Google Cloud CDN cache with ingress controller, Transfer-Encoding header
Kubernetes Cluster-IP service not working as expected
How does rolling update work in kubernetes

1 Answer

3/15/2018

I guess you using something like a minikube for set-up a local Kubernetes cluster and in most of cases it using a virtual machines to spawn a cluster. So, when Kubernetes trying to pull image from localhost address, it connecting to virtual machine local address, not to your computer address. Moreover, your local registry bind only on localhost and not accessible from virtual machines.

The idea of a fix is to make your local docker registry accessible for your Kubernetes and to allow pull images from local insecure registry.

So, first of all, bind your docker registry on your PC to all interfaces:

docker run -d -p 0.0.0.0:5000:5000 --restart=always --name registry registry:2

Then, check your local IP address of the PC. It will be something like 172.X.X.X or 10.X.X.X. The way of the check is depends of your OS, so just google it if you don't know how to get it.

After, start your minikube with an additional option:

minikube start --insecure-registry="<your-local-ip-address>:5000", where a 'your-local-ip-address' is your local IP address.

Now you can try to run a spark job with a new address of a registry and K8s has be able to download your image:

spark.kubernetes.container.image=<your-local-ip-address>:5000/spark:2.3

-- Anton Kostenko
Source: StackOverflow