Deployment.yaml Template file string substitution with sed, all that match a pattern with env var $_<pattern>

11/3/2020

I have a kubernetes deployment template like the following

[...other stuff...]
data:
  ENV_VAR_1: _ENV_VAR_1
  ENV_VAR_2: _ENV_VAR_2
  ENV_VAR_ETC: _ENV_VAR_ETC

I want to instruct my cloudbuild.yaml to replace occurences of _SOME_THING to the actual value of the environment variable $_SOME_THING. I know I can execute Bash commands so I tried this with sed:

# \x27 is the character '
entrypoint: 'bash'
    args:
      - '-eEuo'
      - 'pipefail'
      - '-c'
      - |-
        sed -n 's%\([A-Z][A-Z_0-9]*\):[[:space:]]*_\1[[:space:]]*$%sed -i \
        \x27s;_\1[[:space:]]*$;\x27\$_\1\x27;g\x27 $_K8S_YAML_FILE%gp' \
        $_K8S_YAML_FILE | sh

The idea is generating a substitution commands for each variable and then piping them to shell. The generated commands look good:

sed -i 's;_ENV_VAR_1[[:space:]]*$;'$_ENV_VAR_1';g' $_K8S_YAML_FILE
sed -i 's;_ENV_VAR_2[[:space:]]*$;'$_ENV_VAR_2';g' $_K8S_YAML_FILE
sed -i 's;_ENV_VAR_ETC[[:space:]]*$;'$_ENV_VAR_ETC';g' $_K8S_YAML_FILE

And the whole thing work when I execute it in git bash after setting the env variables with export, but when executing it through the cloudbuild.yaml the file is modified like this:

[...other stuff...]
data: 
  ENV_VAR_1: 1
  ENV_VAR_2: 1
  ENV_VAR_ETC: 1

I think I'm missing some escape or the command line options passed to bash (-eEuo) are messing things up, but I have no clue. Any solution, even not using sed for the task (but requiring no additional tool to install) would be greatly appreciated. Thank you.

-- ilmirons
bash
google-cloud-build
kubernetes
sed

1 Answer

11/6/2020

I came to the conclusion the thing I'm trying to achieve is not possible. Starting a shell via cloudbuild as above and trying to print out env variables (via printenv) and shell variables (set -o posix ; set) does not show any of my variables, that where set as trigger substitution. As far as I know this may actually be what the name says, that is string replacement that are performed on the actual file before it is processed. The evidence is in no way i can pull out the value of those reference if not directly writing them in the file. Any strategy of building a reference and then having it evaluated has failed. Final solution I adopted (that keep the nuisance of having to update the file for every new variable but reads and deal with escapes much better than sed) is the envsubst cloudbuilder (https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/envsubst), propagating each var from file to the shell (env: ['_MY_VAR=$_MYVAR', ...])

-- ilmirons
Source: StackOverflow