I have a Jenkins pod running in GCP's Kubernetes Engine and I'm trying to run a maven unit test that connects to a google cloud SQL database to perform said test. My application.yaml for my project looks like this:

spring:
  cloud:
     gcp:
        project-id: <my_project_id>
     sql: 
        database-name: <my_database_name>
        instance-connection-name: <my_instance_connection_name>

  jpa:
     database-platform: org.hibernate.dialect.MySQL55Dialect
     hibernate:
        ddl-auto: create-drop

  datasource:
     continue-on-error: true
     driver-class-name: com.mysql.cj.jdbc.Driver
     username: <my_cloud_sql_username>
     password: <my_cloud_sql_password>

The current Jenkinsfile associated with this project is:

Jenkinsfile:

pipeline {
   agent any

   tools{
      maven 'Maven 3.5.2'
      jdk 'jdk8'
   }
   environment {
       IMAGE = readMavenPom().getArtifactId()
       VERSION = readMavenPom().getVersion()
       DEV_DB_USER = "${env.DEV_DB_USER}"
       DEV_DB_PASSWORD = "${env.DEV_DB_PASSWORD}"
    }

   stages {
      stage('Build docker image') {
        steps {
            sh 'mvn -Dmaven.test.skip=true clean package'
            script{
               docker.build '$IMAGE:$VERSION'
            }

        }
    }
       stage('Run unit tests') {

          steps {
            withEnv(['GCLOUD_PATH=/var/jenkins_home/google-cloud-sdk/bin']) {
                withCredentials([file(credentialsId: 'key-sa', variable: 'GC_KEY')]) {
                   sh("gcloud auth activate-service-account --key-file=${GC_KEY}")
                   sh("gcloud container clusters get-credentials <cluster_name> --zone northamerica-northeast1-a --project <project_id>")
                   sh 'mvn test'
                }
             }          
           }                
        }
    }
}

}

My problem is when the Pipeline actually tries to run the mvn test using the above configuration (in my application.yaml) I'm getting this error:

  Caused by: 
  com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 
  Forbidden
  {
     "code" : 403,
     "errors" : [ {
        "domain" : "global",
        "message" : "Insufficient Permission: Request had insufficient authentication scopes.",
        "reason" : "insufficientPermissions"
      } ],
     "message" : "Insufficient Permission: Request had insufficient authentication scopes."
  }

I have two Google Cloud projects:

• One that has the Kubernetes Cluster where the Jenkins pod is running.

• Another project where the K8s Cluster contains my actual Spring Boot Application and the Cloud SQL database that I'm trying to access.

I also created the service account only in my Spring Boot Project for Jenkins to use with three roles: Cloud SQL Editor, Kubernetes Engine Cluster Admin and Project owner (to verify that the service account is not at fault).

I enabled the Cloud SQL, Cloud SQL admin and Kubernetes APIs in both projects and I double checked my Cloud SQL credentials and they are ok. In addition, I authenticated the Jenkins pipeline using the json file generated when I created the service account, following the recommendations discussed here:

Jenkinsfile (extract):

  ...
  withCredentials([file(credentialsId: 'key-sa', variable: 'GC_KEY')]) {
               sh("gcloud auth activate-service-account --key-file=${GC_KEY}")
               sh("gcloud container clusters get-credentials <cluster_name> --zone northamerica-northeast1-a --project <project_id>")
               sh 'mvn test'
   }
   ...