It's pretty standard spark-submit action, what's weird is that I got 401 from time to time, if I just wait a few minute, I can run again, until next time I got a 401.
it's very similar to this issue https://jira.apache.org/jira/browse/SPARK-24227
I'm just wondering if I'm not setting the GOOGLE_APPLICATION_CREDENTIAL right when submitting, but at the very beginning I haven't set any credential and can still run.
Anyone has similar issues?
It's a subtle bug with Spark couldnt refresh the access token. One work around is: