I'm trying to run PySpark on a Kubernetes cluster on AWS.

I'm submitting to the cluster with the spark-submit command and viewing the results in the Kubernetes dashboard.

The driver pod is getting created fine, but the executors frequently fail to spin up, failing with either of the following errors:

Failed to pull image "docker.io/joemalt/[image-name]:[tag]": rpc error: code = Unknown desc = Error response from daemon: unauthorized: authentication required

Failed to pull image "docker.io/joemalt/[image name]:[tag]": rpc error: code = Unknown desc = Error response from daemon: error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"

Kubernetes attempts to recreate the pods, but the errors are frequent enough that it often doesn't manage to get any executor pods working at all.

Neither of these errors occur when setting up the driver pod, or when pulling the image manually. The repository is public so the authentication required in particular doesn't make any sense to me. I've tried replacing the Kubernetes cluster, with no success.