Basically, when using Google Cloud Build, how do I read a value that was written in an earlier build step in subsequent steps?
Specifically, I'd like to make a custom image tag that's based on a combination of the timestamp and $SHORT_SHA. Something like the below. Though, it doesn't work, as docker complains about "export", and, even if that worked, it likely will be a different env:
# Setting tag in a variable:
- name: 'ubuntu'
args: ['export', '_BUILD_TAG=`date', '-u', '+%Y%m%dT%H%M%S_$SHORT_SHA`']
Then, in a later step:
# Using tag from the variable:
- name: gcr.io/cloud-builders/docker
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$_BUILD_TAG', '.']
So, how do I use the output of one step in another? I could write the contents of date
to a file, and then read it, but I'm back at not knowing how to set the variable from the file I read (or otherwise interpolate its results to form the argument to docker build).
You don't need to export nor mount a volume in your case.
steps:
- name: 'ubuntu'
entrypoint: 'bash'
args:
- '-c'
- |
printenv
- name: gcr.io/cloud-builders/docker
entrypoint: 'bash'
args:
- '-c'
- |
printenv
It will output
BUILD
Starting Step #0
Step #0: Pulling image: ubuntu
Step #0: Using default tag: latest
Step #0: latest: Pulling from library/ubuntu
Step #0: Digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
Step #0: Status: Downloaded newer image for ubuntu:latest
Step #0: HOSTNAME=XXXXXXXXXXX
Step #0: BUILDER_OUTPUT=/builder/outputs
Step #0: PWD=/workspace
Step #0: HOME=/builder/home
Step #0: SHLVL=1
Step #0: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Step #0: _=/usr/bin/printenv
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: HOSTNAME=XXXXXXXXXXX
Step #1: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Step #1: PWD=/workspace
Step #1: SHLVL=1
Step #1: HOME=/builder/home
Step #1: DEBIAN_FRONTEND=noninteractive
Step #1: BUILDER_OUTPUT=/builder/outputs
Step #1: _=/usr/bin/printenv
Finished Step #1
So you can use /workspace
or /builder/home
, but since we cannot use a variable other than the defined substitution on yaml file then put them as script in the repo like this:
steps:
- name: 'ubuntu'
entrypoint: 'bash'
args:
- '-c'
- |
bash test.bash
- name: gcr.io/cloud-builders/docker
entrypoint: 'bash'
args:
- '-c'
- |
bash result.bash
test.bash
#!/bin/bash
SHORT_SHA=myvar
date -u +%Y%m%dT%H%M_$SHORT_SHA > /workspace/myfile.txt
result.bash
#!/bin/bash
_BUILD_TAG=`cat /workspace/myfile.txt`
echo "the transferred value is: $_BUILD_TAG"
Output:
BUILD
Starting Step #0
Step #0: Pulling image: ubuntu
Step #0: Using default tag: latest
Step #0: latest: Pulling from library/ubuntu
Step #0: Digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
Step #0: Status: Downloaded newer image for ubuntu:latest
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: the transferred value is: 20190708T1706_myvar
Finished Step #1
PUSH
DONE
Too bad this isn't supported (yet) by Google. However, I have been using the following simple method a lot and it works fine. Keep in mind that the file is saved to /workspace
by default, which is shared among containers. If you would need it in a different directory, save or copy it somewhere else.
# Save variable to file
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
_id=$(openssl rand -hex 16,,)
echo "${_id}" > id.txt
# Set variable from file
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
_id=$(cat id.txt)
echo "${_id}"
I never found a way to set an environment variable in one build step that can be read in other steps, but I ended up accomplishing the same effect by building on Konstantin's answer in the following way:
In an early step, I generate and write my date-based tag to a file. The filesystem (/workspace) is retained between steps, and serves as store of my environment variable. Then, in each step that I need to reference that value, I cat that file in place. The trick is to use sh or bash as the entrypoint in each container so that the sub-shell that reads from the file can execute.
Here's an example:
## Set build tag and write to file _TAG
- name: 'ubuntu'
args: ['bash', '-c', 'date -u +%Y%m%dT%H%M_$SHORT_SHA > _TAG']
...
# Using the _TAG during Docker build:
- name: gcr.io/cloud-builders/docker
entrypoint: sh
args: ['-c', 'docker build -t gcr.io/$PROJECT_ID/image_name:$(cat _TAG) .']
A caveat to note is that if you are doing the bash interpolation in this way within, say, a JSON object or something that requires double quotes, you need the subshell call to never be surrounded by single quotes when executed in the container, only double, which may require escaping the internal double quotes to build the JSON object. Here's an example where I patch the kubernetes config using the _TAG file value to deploy the newly-build image:
- name: gcr.io/cloud-builders/kubectl
entrypoint: bash
args: ['-c', 'gcloud container clusters get-credentials --zone $CLOUDSDK_COMPUTE_ZONE $CLOUDSDK_CONTAINER_CLUSTER ; kubectl patch deployment deployment_name -n mynamespace -p "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"image_name\",\"image\":\"gcr.io/$PROJECT_ID/image_name:$(cat _TAG)\"}]}}}}}"']
env:
- 'CLOUDSDK_COMPUTE_ZONE=us-central1-b'
- 'CLOUDSDK_CONTAINER_CLUSTER=my-google-proj-cluster-name'
- name: gcr.io/cloud-builders/docker
entrypoint: sh
args
- '-c'
- 'docker build -t gcr.io/$PROJECT_ID/$(date -u +%Y%m%dT%H%M%S_$SHORT_SHA) .'
Although this doesn't solve your problem, I did want to post this answer since the first sentence of your question is, "Basically, when using Google Cloud Build, how do I read a value that was written in an earlier build step in subsequent steps?". This is how you'd do that.
From the official documentation:
A Volume is a Docker container that is mounted into build steps to persist files across build steps. When Cloud Build runs a build step, it automatically mounts a workspace volume into /workspace. You can specify additional volumes to be mounted into your build steps' containers using the volumes field for your steps.
Here's an example of it implemented from someone who asked this question in a github issue, but to put the date in the volume for later reading by another step:
steps:
- name: 'ubuntu'
volumes:
- name: 'vol1'
path: '/persistent_volume'
entrypoint: 'bash'
args:
- '-c'
- |
date -u +%Y%m%dT%H%M_$SHORT_SHA > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
volumes:
- name: 'vol1'
path: '/persistent_volume'
args: ['run', '-v', 'vol1:/data', 'alpine', 'cat', 'data/file']
However, for your particular case, I would just tag it with a subshell command like it's done in this answer here:
$(date -u +%Y%m%dT%H%M%S_$SHORT_SHA)
Here's an example of what I've just done myself to reuse output from GitVersion in another. It builds on the answer @chetabahana posted.
steps:
- id: 'Gitversion: Unshallow repo'
name: gcr.io/cloud-builders/git
args: [fetch, --unshallow]
- id: 'Gitversion: Parse'
name: gittools/gitversion:latest-linux
entrypoint: /bin/bash
args:
- -c
- |
dotnet /app/GitVersion.dll > /workspace/gitversion.json
- id: 'Gitversion: Env file'
name: stedolan/jq
entrypoint: /bin/bash
args:
- -c
- |
for s in $(cat /workspace/gitversion.json | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
echo "export $s" >> /workspace/gitversion.env
done
- id: 'Build and push API image'
name: gcr.io/cloud-builders/docker
entrypoint: /bin/bash
args:
- -c
- |
source /workspace/gitversion.env
docker build -t gcr.io/xxxx/example:$${SemVer}-$${BuildMetaData} example-app
docker push gcr.io/xxxx/example:$${SemVer}-$${BuildMetaData}
The magic ingredient was the $
to escape substitution variable so that the build job would not attempt to substitute, leaving it for bash
to substitute instead.