Parallelizing google cloudbuild steps without mangling the build logs

4/3/2019

Following guidance here: https://cloud.google.com/cloud-build/docs/configuring-builds/configure-build-step-order

We have split up our build into a multilayer docker image where layers are

  1. Install OS and third party deps
  2. Install our sources and build (in debug or release depending on docker ARG)
  3. Run ci / code coverage

We have tagged the steps in cloudbuild.yaml with docker id's and are using waitfor to try to make it so that the debug and release versions can run in parallel.

However, when we do this, the build log is mixed up -- the build logs for release and debug are are jumbled together making it much harder to read. e.g.

Step #2 - "build-debug": �[0m�[91m  Downloaded colored v1.7.0
Step #5 - "build-release": �[0m�[91merror: couldn't read /tmp/mobilenode/src/attest/src/ias/../data/AttestationReportSigningCACert.pem: No such file or directory (os error 2)
Step #5 - "build-release":   --> /tmp/mobilenode/src/attest/src/ias/verify.rs:35:7
Step #5 - "build-release":    |
Step #5 - "build-release": 35 |     &[include_str!("../data/AttestationReportSigningCACert.pem")];
Step #5 - "build-release":    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Step #5 - "build-release": 
Step #2 - "build-debug": �[0m�[91m  Downloaded toml v0.4.10
[0m�[91m  Downloaded safemem v0.3.0
Step #2 - "build-debug": �[0m�[91m  Downloaded crunchy v0.1.6
Step #2 - "build-debug": �[0m�[91m  Downloaded grpcio-sys v0.4.4
Step #5 - "build-release": �[0m�[91merror: aborting due to previous error
Step #5 - "build-release": 
Step #5 - "build-release": �[0m�[91merror: Could not compile `attest`.
Step #5 - "build-release": warning: build failed, waiting for other jobs to finish...
Step #5 - "build-release": �[0m�[91merror: build failed
Step #5 - "build-release": �[0m�[91mmake: *** [src/enclave/target/release/libenclave.so] Error 101
Step #5 - "build-release": �[0mMakefile:90: recipe for target 'src/enclave/target/release/libenclave.so' failed
Step #2 - "build-debug": �[0m�[91m  Downloaded term v0.5.1
Step #2 - "build-debug": �[0m�[91m  Downloaded tiny_http v0.6.2
Step #2 - "build-debug": �[0m�[91m  Downloaded regex v0.1.80

Is there a way to configure cloudbuild so that there is a separate log file for each build step? Is the best answer to use a different cloudbuild.yaml for the parallel steps, and skip all this wait_for stuff?

-- Chris Beck
continuous-integration
continuous-testing
google-cloud-build
google-cloud-platform
google-kubernetes-engine

0 Answers