K
Q

Google Cloud Kubernetes not reading application credentials correctly

8/8/2018

I'm trying to host an application using Google Kubernetes Engine. My docker image works when run locally, but when I put it on Google Cloud and set it up using a kubernetes cluster it fails in a very strange way.

I'm able to connect to the application and it works until I trigger a call of google.cloud.storage.Client(). Then it attempts to read the file I've provided through the GOOGLE_APPLICATION_CREDENTIALS environment variable and something goes wrong. I get this (truncated and redacted) traceback:

self.gcs_client = storage.Client()
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/client.py", line 71, in __init__
    _http=_http)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line 215, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line 169, in __init__
    project = self._determine_default(project)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/client.py", line 182, in _determine_default
    return _determine_default_project(project)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 294, in default
    credentials, project_id = checker()
  File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 165, in _get_explicit_environ_credentials
    os.environ[environment_vars.CREDENTIALS])
  File "/usr/local/lib/python3.6/site-packages/google/auth/_default.py", line 89, in _load_credentials_from_file
    'File {} was not found.'.format(filename))
google.auth.exceptions.DefaultCredentialsError: File {--redacted--} was not found.

What I've redacted is JSON containing my service account's private key. I've checked the docker container while it's running on the cloud and GOOGLE_APPLICATION_CREDENTIALS is set to a file name as expected. Somehow when my docker container is run on the cloud, instead of using the environment variable as the file name - it uses the contents of the referenced file as the file name. This error also gets reported in the browser console, so anyone that navigates to the app can get my service account credentials.

Does anyone have any guesses about what's going wrong here?

UPDATE: Looks like I've got it working now. My guess is the error occurred because I set GOOGLE_APPLICATION_CREDENTIALS using valueFrom as in the bokeh.yaml file. This seems to have been setting GOOGLE_APPLICATION_CREDENTIALS equal to the contents of bokeh.yaml. This format seems to work for pandas.io.gbq.read_gbq used in the tutorial code but not for instantiating google.cloud.storage.Client() like I was trying to do. Pointing GOOGLE_APPLICATION_CREDENTIALS to a volume mounted file like DazWilkin suggested and Eric Guan showed worked.

-- hamdog
docker
google-cloud-platform
google-cloud-storage
kubernetes
python

Similar Questions

Minikube running out of space and failing despite --disk-size flag
Which CNI plugins are supported
nginx ingress Jenkins path rewrite configuration not working
Cloud SQL connection from Kubernetes in Go - Error 403: Insufficient Permission

1 Answer

8/9/2018

I noticed you didn't mention using Kubernetes Secrets. Here's how I did it.

  1. Create a Secret with kubectl on your local machine.
$ kubectl create secret generic gac-keys --from-file=<PATH_TO_SERVICE_ACCOUNT_FILE>

This creates a Secret called gac-keys. It contains your json file found at <PATH_TO_SERVICE_ACCOUNT_FILE>. If you want to rename the json file, you can do

--from-file=new-file-name.json=<PATH_TO_SERVICE_ACCOUNT_FILE>

  1. Configure your Deployment to use the Secret.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      volumes:
      - name: google-cloud-keys
        secret:
          secretName: gac-keys
      containers:
      - name: my-app
        image: us.gcr.io/my-app
        volumeMounts:
        - name: google-cloud-keys
          mountPath: /var/secrets/google
          readOnly: true
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/new-file-name.json

You specify a volume with an arbitrary name google-cloud-keys to be referenced later. The volume is linked to the secret. Inside the container spec, you mount the google-cloud-keys volume at path /var/secrets/google. This places your file at the path, so /var/secrets/google/new-file-name.json should exist in the container at runtime. Then you specify an env variable named GOOGLE_APPLICATION_CREDENTIALS which points to the path. Now your client library can authenticate with google.

Docs: https://kubernetes.io/docs/concepts/configuration/secret/

-- Eric Guan
Source: StackOverflow