I am attempting to upload a file to a Google cloud storage bucket from a service running on a Kubernetes pod. This requires authentication for the Google storage and so I have created a json authentication file from the console.
This json file is saved as a secret on my kubernetes environment and is referenced in the deploy.yaml through the environment variable GOOGLE_APPLICATION_CREDENTIALS.
The format appears like this:
{
"type": "service_account",
"project_id": "xxx",
"private_key_id": "xxx",
"private_key": "-----BEGIN PRIVATE KEY-----\nxxx\n-----END PRIVATE KEY-----\n",
"client_email": "xx",
"client_id": "",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "xxx"
}
And the following code of how I am handling this authentication:
import google.auth
credentials, project_id = google.auth.default()
bucket_name = Config.STORAGE_BUCKET_NAME
client = storage.Client(project=project_id)
bucket = client.get_bucket(bucket_name)
Given that locally I have logged into gcloud, I am able to upload files when testing locally. However, when deployed to Kubernetes I get the following error:
File "/storage.py", line 8, in <module>
credentials, project_id = google.auth.default()
File "/usr/local/lib/python3.6/dist-packages/google/auth/_default.py", line 308, in default
credentials, project_id = checker()
File "/usr/local/lib/python3.6/dist-packages/google/auth/_default.py", line 166, in _get_explicit_environ_credentials
os.environ[environment_vars.CREDENTIALS]
File "/usr/local/lib/python3.6/dist-packages/google/auth/_default.py", line 92, in _load_credentials_from_file
"File {} was not found.".format(filename)
google.auth.exceptions.DefaultCredentialsError: File {
"type": "service_account",
"project_id": "xxx",
"private_key_id": "xxx",
"private_key": "-----BEGIN PRIVATE KEY-----\nxxx\n-----END PRIVATE KEY-----\n",
"client_email": "xx",
"client_id": "",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "xxx"
}
was not found.
I'm confused as it's referencing the file itself in the error so I am assuming it located it, but it does not recognize it as a valid service authentication file.
All help and pointers appreciated. :)
My issue ended up being that I had no set up the secret json key file as a volume mount in the deploy file.
Example of how I did it:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /mnt/file_key/key.json
.......
volumeMounts:
- mountPath: /mnt/file_key
name: file_key
readOnly: true
........
volumes:
- name: file_key
secret:
defaultMode: 420
secretName: file_key_secretI think you should try this code locally and then on kubernetes, to check if is a problem with your service_account.json file:
client = storage.Client.from_service_account_json(
'service_account.json') or link:
If your application runs on Compute Engine, Kubernetes Engine, the App Engine flexible environment, or Cloud Functions, you don't need to create your own service account. Compute Engine includes a default service account that is automatically created for you, and you can assign a different service account, per-instance, if needed.
credentials = compute_engine.Credentials()
# Create the client using the credentials and specifying a project ID.
storage_client = storage.Client(credentials=credentials, project=project)