Can't use Google Cloud Kubernetes substitutions

10/25/2019

Obviously, I'm doing something in the wrong way, but I can't understand where is the problem. I'm a new to Kubernetes.

There is the Node.js app, I'm able to wrap it to the Docker and deploy to Google Compute engine (it works with Git trigger and locally). The most important thing here - there are env variables, some of them are secret, encrypted with key. Google uses it too to decrypt values and give them to the application during the build process (everything is done based on Google docs). Now I'm trying to change cloudbuild.yaml file to get Kubernetes config.

cloudbuild.yaml (part of settings may be redundant after switching from Docker to Kubernetes). Without marked section below in cloudbuild.yaml I'm getting the following error:

Error merging substitutions and validating build: Error validating build: key "_DB_HOST" in the substitution data is not matched in the template;key "_STATIC_SECRET" in the substitution data is not matched in the template;key "_TYPEORM_DATABASE" in the substitution data is not matched in the template;key "_TYPEORM_PASSWORD" in the substitution data is not matched in the template;key "_TYPEORM_USERNAME" in the substitution data is not matched in the template Blockquote

which is correct because Google considers unused substitutions as errors. But if I leave marked section I'm getting this error:

Error merging substitutions and validating build: Error validating build: invalid .secrets field: secret 0 defines no secretEnvs

which is totally unclear for me.

cloudbuild file:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args: [
      '-c',
      'docker pull gcr.io/$PROJECT_ID/myproject:latest || exit 0'
    ]
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t',
      'gcr.io/$PROJECT_ID/myproject:$BRANCH_NAME-$COMMIT_SHA',
      '-t',
      'gcr.io/$PROJECT_ID/myproject:latest',
# <<<<<------- START OF DESCRIBED SECTION
      'DB_HOST=${_DB_HOST}',
      'TYPEORM_DATABASE=${_TYPEORM_DATABASE}',
      'TYPEORM_PASSWORD=${_TYPEORM_PASSWORD}',
      'TYPEORM_USERNAME=${_TYPEORM_USERNAME}',
      'STATIC_SECRET=${_STATIC_SECRET}',
# <<<<<------- END OF DESCRIBED SECTION
      '.'
    ]
  - name: 'gcr.io/cloud-builders/kubectl'
    args: [ 'apply', '-f', '/' ]
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=<region>'
      - 'CLOUDSDK_CONTAINER_CLUSTER=myproject'
  - name: 'gcr.io/cloud-builders/kubectl'
    args: [
      'set',
      'image',
      'deployment',
      'myproject',
      'myproject=gcr.io/$PROJECT_ID/myproject:$BRANCH_NAME-$COMMIT_SHA'
    ]
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=<region>'
      - 'CLOUDSDK_CONTAINER_CLUSTER=myproject'
      - 'DB_PORT=5432'
      - 'DB_SCHEMA=public'
      - 'TYPEORM_CONNECTION=postgres'
      - 'FE=myproject'
      - 'V=1'
      - 'CLEAR_DB=true'
      - 'BUCKET_NAME=myproject'
      - 'BUCKET_TYPE=google'
      - 'KMS_KEY_NAME=storagekey'
    secretEnv:
      - DB_HOST,
      - TYPEORM_DATABASE,
      - TYPEORM_PASSWORD,
      - TYPEORM_USERNAME,
      - STATIC_SECRET
timeout: 1600s
substitutions:
  _DB_HOST: $DB_HOST
  _TYPEORM_DATABASE: $TYPEORM_DATABASE
  _TYPEORM_PASSWORD: $TYPEORM_PASSWORD
  _TYPEORM_USERNAME: $TYPEORM_USERNAME
  _STATIC_SECRET: $STATIC_SECRET
secrets:
  - kmsKeyName: projects/myproject/locations/global/keyRings/storage/cryptoKeys/storagekey
  - secretEnv:
      DB_HOST: <encrypted base64 here>
      TYPEORM_DATABASE: <encrypted base64 here>
      TYPEORM_PASSWORD: <encrypted base64 here>
      TYPEORM_USERNAME: <encrypted base64 here>
      STATIC_SECRET: <encrypted base64 here>
images:
  - 'gcr.io/$PROJECT_ID/myproject:$BRANCH_NAME-$COMMIT_SHA'
  - 'gcr.io/$PROJECT_ID/myproject:latest'

secret.yaml file (registered in kubectl as it should be):

apiVersion: v1
kind: Secret
metadata:
  name: myproject
type: Opaque
data:
  DB_HOST: <encrypted base64 here>
  TYPEORM_DATABASE: <encrypted base64 here>
  TYPEORM_PASSWORD: <encrypted base64 here>
  TYPEORM_USERNAME: <encrypted base64 here>
  STATIC_SECRET: <encrypted base64 here>

pod.yaml file

apiVersion: v1
kind: Pod
metadata:
  name: myproject
spec:
  containers:
    - name: myproject
      image: gcr.io/myproject/myproject:latest
      # project ID is valid here, don't bother on mock values
      env:
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: myproject
              key: DB_HOST
        - name: TYPEORM_DATABASE
          valueFrom:
            secretKeyRef:
              name: myproject
              key: TYPEORM_DATABASE
        - name: TYPEORM_PASSWORD
          valueFrom:
            secretKeyRef:
              name: myproject
              key: TYPEORM_PASSWORD
        - name: TYPEORM_USERNAME
          valueFrom:
            secretKeyRef:
              name: myproject
              key: TYPEORM_USERNAME
        - name: STATIC_SECRET
          valueFrom:
            secretKeyRef:
              name: myproject
              key: STATIC_SECRET
    restartPolicy: Never
-- WeekendMan
docker
google-cloud-platform
google-kubernetes-engine
kubernetes

1 Answer

10/25/2019

I think, you mix too many things, your legacy build and your new one. If your secrets are already set in your cluster, you don't need them at the build time.

Try this, with only the required step for deploying (no substitution, no secret, no KMS)

steps:
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args: [
      '-c',
      'docker pull gcr.io/$PROJECT_ID/myproject:latest || exit 0'
    ]
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t',
      'gcr.io/$PROJECT_ID/myproject:$BRANCH_NAME-$COMMIT_SHA',
      '-t',
      'gcr.io/$PROJECT_ID/myproject:latest',
      '.'
    ]
  - name: 'gcr.io/cloud-builders/kubectl'
    args: [ 'apply', '-f', '/' ]
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=<region>'
      - 'CLOUDSDK_CONTAINER_CLUSTER=myproject'
  - name: 'gcr.io/cloud-builders/kubectl'
    args: [
      'set',
      'image',
      'deployment',
      'myproject',
      'myproject=gcr.io/$PROJECT_ID/myproject:$BRANCH_NAME-$COMMIT_SHA'
    ]
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=<region>'
      - 'CLOUDSDK_CONTAINER_CLUSTER=myproject'
      - 'DB_PORT=5432'
      - 'DB_SCHEMA=public'
      - 'TYPEORM_CONNECTION=postgres'
      - 'FE=myproject'
      - 'V=1'
      - 'CLEAR_DB=true'
      - 'BUCKET_NAME=myproject'
      - 'BUCKET_TYPE=google'
      - 'KMS_KEY_NAME=storagekey'
timeout: 1600s
images:
  - 'gcr.io/$PROJECT_ID/myproject:$BRANCH_NAME-$COMMIT_SHA'
  - 'gcr.io/$PROJECT_ID/myproject:latest
-- guillaume blaquiere
Source: StackOverflow