Kubeflow pipelines can't fetch model from google cloud storage

11/15/2021

I've set up Kubeflow on GCP following instructions from https://www.kubeflow.org/docs/distributions/gke/deploy/

In the same project (as Kubeflow cluster) I store a model in a bucket which I attach to KFServing's InferenceService. This is a part of a kubeflow pipeline which is run using default-editor KSE. Container storage-initializer for pod which serves the model crashes because it receives Forbidden error with message Caller does not have storage.objects.list access to the Google Cloud Storage bucket. when trying to fetch the model (Notice that for non-authenticated users the message would say ''Anonymous caller...'').

This is the output of kubectl describe serviceaccount default-editor

Name:                default-editor
Namespace:           NAMESPACE
Labels:              <none>
Annotations:         iam.gke.io/gcp-service-account: kubeflow-user@PROJECT.iam.gserviceaccount.com
Image pull secrets:  pipelines-secret
Mountable secrets:   default-editor-token-XXXX
Tokens:              default-editor-token-XXXX
Events:              <none>

This service account has been granted 'editor' role:

gcloud projects add-iam-policy-binding PROJECT \
  --member="serviceAccount:kubeflow-user@PROJECT.iam.gserviceaccount.com" \
  --role="roles/editor"

Do you have any idea why I receive such an error?

-- Tomasz Cakala
authentication
google-cloud-platform
kubeflow
kubeflow-pipelines
kubernetes

0 Answers