I am running Kubeflow pipeline(docker approach) and cluster uses the endpoint to navigate to the dashboard. The Clusters is created followed by the instructions mentioned in this link Deploy Kubeflow. Everything is successfully created and the cluster generated the endpoints and its working perfectly.

Endpoint link would be something like this https://appname.endpoints.projectname.cloud.goog.

Every workload of the pipeline is working fine except the last one. In the last workload, I am trying to submit a job to the cloud-ml engine. But it logs shows that the application has no access to the project. Here is the full image of the log.

ERROR: (gcloud.ml-engine.versions.create) PERMISSION_DENIED: Request had insufficient authentication scopes.

ERROR: (gcloud.ml-engine.jobs.submit.prediction) User [clustername@project_name.iam.gserviceaccount.com] does not have permission to access project [project_name] (or it may not exist): Request had insufficient authentication scopes.

From the logs, it's clear that this service account doesn't have access to the project itself. However, I tried to give access for Cloud ML Service to this service account but still, it's throwing the same error.

Any other ways to give Cloud ML service credentials to this application.