spark-submit: 403 error, client System:anonymous error

11/19/2018

From time to time, when I submit spark job to Google Kubernetes cluster, I got 401 unauthorized, so I do this gcloud container clusters get-credential my-cluster, but it almost always followed by 403 error, saying client system:anonymous etc., but the weird the part is, I just need to do a simple kubectl get namespace, then it works again, why is that?

-- dex
apache-spark
google-kubernetes-engine
kubernetes
spark-submit

1 Answer

11/19/2018

From time to time, when I submit spark job to Google Kubernetes cluster, I got 401 unauthorized

Your credentials expire.

I do this gcloud container clusters get-credential my-cluster, but it almost always followed by 403 error, saying client system:anonymous etc.

Sounds like a timing issue where your new credentials and getting propagated in the cluster.

-- Rico
Source: StackOverflow