I have been struggling to push a docker image to a Google Container Registry using fabric8 maven plugin on a Jenkins pipeline. I have checked every question on stackoverflow but none of them solved my problem.
This is my setup:
Kubernetes Cluster running on Google Kubernetes Engine. I have deployed a pod with a Jenkins server that starts agents with the Kubernetes CI plugin, based on this custom image:
FROM openjdk:8
RUN apt-get update && \
apt-get -y install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable" && \
apt-get update && \
apt-get -y install docker-ce
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
This agent has Java 8 and Docker.
I want to use fabric8 maven plugin to perform the docker image build and push to Google Container Registry. The point is no matter what I do I always end up facing this error (unauthorized):
Unable to push 'eu.gcr.io/myprojectid/gke-springboot-sample' from registry 'eu.gcr.io' : unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication -> [Help 1]
NOTE:
I have created a service account associated to my GC project and given it these permissions: project editor and storage admin.
I have configured a Global Jenkins Secret (cloud-registry-credentials) associated to the key.json file belonging to that service account
There are all the things I have tried in my pipeline so far:
1: oauth2accesstoken
stage("Build docker image") {
agent {
label 'jenkins-slave'
}
steps {
container('jenkins-slave'){
script {
sh "./mvnw clean build fabric8:build"
withCredentials([file(credentialsId: 'cloud-registry-credentials', variable: 'crc')]) {
sh "gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://eu.gcr.io"
sh "./mvnw fabric8:push"
}
}
}
}
}
OUTPUT:
+ gcloud auth print-access-token
+ docker login -u oauth2accesstoken --password-stdin https://eu.gcr.io
WARNING! Your password will be stored unencrypted in /home/jenkins/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[Pipeline] sh
Running shell script
+ ./mvnw fabric8:push
[INFO] F8> The push refers to a repository [eu.gcr.io/myprojectid/gke-springboot-sample]
#
[ERROR] F8> Unable to push 'eu.gcr.io/projectid/gke-springboot-sample' from registry 'eu.gcr.io' : unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication [unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication ]
2: key.json
stage("Build docker image") {
agent {
label 'jenkins-slave'
}
steps {
container('jenkins-slave'){
script {
sh "./mvnw fabric8:build"
withCredentials([file(credentialsId: 'cloud-registry-credentials', variable: 'crc')]) {
sh "docker login -u _json_key --password-stdin https://eu.gcr.io < ${crc}"
sh "gcloud auth activate-service-account --key-file=${crc}"
sh "./mvnw fabric8:push"
}
}
}
}
}
OUTPUT:
+ docker login -u _json_key --password-stdin https://eu.gcr.io
WARNING! Your password will be stored unencrypted in /home/jenkins/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[Pipeline] sh
Running shell script
+ gcloud auth activate-service-account --key-file=****
Activated service account credentials for: [jenkins-cr@myprojectid.iam.gserviceaccount.com]
[Pipeline] sh
Running shell script
+ ./mvnw fabric8:push
[ERROR] F8> Unable to push 'eu.gcr.io/myprojectid/gke-springboot-sample' from registry 'eu.gcr.io' : unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication [unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication ]
After adding the -X flag to the maven ./mvnw fabric8:push
command I found the following DEBUG line: AuthConfig: no credentials found
.
I had to dig in the source code of the AuthConfigFactory class to find out that the problem was that it didn't find the previously generated /home/jenkins/.docker/config.json
file.
The problem is that this container's home is /root so it was looking for it in /root/.docker/config.json
and it wasn't there. As a workaround I added sh "cp /home/jenkins/.docker/config.json /root/.docker/config.json"
and now it works like a charm:
[DEBUG] F8> AuthConfig: credentials from ~.docker/config.json
[INFO] F8> myprojectid/gke-springboot-sample:0.0.1-SNAPSHOTsnapshot-180813-080943-0179 in 5 seconds
you might be looking for the Jenkins Google Container Registry Auth Plugin.
another possible reason may be, that the user jenkins-cr
does not have the required role storage.admin
assigned, in Cloud IAM... here's the relevant documentation. just seen you've assigned it; maybe double-check that.
you might also be able to get support there, in case this is rather related to the fabric8 plugin.
also, hence it is a bucket, that bucket's ACL
could eventually interfere.