Kubernetes - How to setup parallels clusters (Prod, Dev) sharing the same repositories/pipeline

2/17/2020

I am currently facing a situation where I need to deploy a small cluster (only 4 pods for now) which will be containing 4 different microservices. This cluster has to be duplicated so I can have one PRODUCTION cluster and one DEVELOPMENT cluster.

Even if it's not hard from my point of view (Creating a cluster and then uploading docker images to pods with parameters in order to use the right resources connection strings), I am stuck at the CD/CI part..

From a CloudBuild trigger, how to push the Docker image to the right "cluster's pod", I have absolutely no idea AT ALL how to achieve it...

There is my cloudbuild.yaml

steps:
  #step 1 - Getting the previous (current) image
  - name: 'gcr.io/cloud-builders/docker'
    entrypoint: 'bash'
    args: [
      '-c',
      'docker pull gcr.io/{PROJECT_ID}/{SERVICE_NAME}:latest || exit 0'
    ]
  #step 2 - Build the image and push it to gcr.io
  - name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t',
      'gcr.io/{PROJECT_ID}/{SERVICE_NAME}',
      '-t',
      'gcr.io/{PROJECT_ID}/{SERVICE_NAME}:latest',
      '.'
    ]
  #step 3 - Deploy our container to our cluster
  - name: 'gcr.io/cloud-builders/kubectl'
    args: ['apply', '-f', 'service.yaml', '--force']
    env:
      - 'CLOUDSDK_COMPUTE_ZONE={CLUSTER_ZONE}'
      - 'CLOUDSDK_CONTAINER_CLUSTER={CLUSTER_NAME}'
  #step 4 - Set the image
  - name: 'gcr.io/cloud-builders/kubectl'
    args: [
      'set',
      'image',
      'deployment',
      '{SERVICE_NAME}',
      '{SERVICE_NAME}=gcr.io/{PROJECT_ID}/{SERVICE_NAME}'
    ]
    env:
      - 'CLOUDSDK_COMPUTE_ZONE={CLUSTER_ZONE}'
      - 'CLOUDSDK_CONTAINER_CLUSTER={CLUSTER_NAME}'
# push images to Google Container Registry with tags
images: [
  'gcr.io/{PROJECT_ID}/{SERVICE_NAME}',
  'gcr.io/{PROJECT_ID}/{SERVICE_NAME}:latest'
]

Can anyone help me out? I don't really know in which direction to go to..

-- Emixam23
docker
google-cloud-build
google-cloud-platform
kubernetes

3 Answers

2/23/2020

The short answer for this is to apply GitOps deployment practices in your workflow.

  1. All your Kubernetes YAMLs or Helmcharts are in a single git repository which is used by a GitOps operator.
  2. In your CI pipeline, you only have to build and push docker images.
  3. The GitOps operator intelligently fetches images versions and make changes and a commit to the target Kubernetes YAML in the repository.
  4. Every change in the GitOps repository is applied to the cluster.

See https://fluxcd.io/

-- john
Source: StackOverflow

2/18/2020

Did you know helm chart? It is designed for different environment deployment.

With different values.yaml file, you can quickly deploy to different environment with same source code base.

For example, you can name the values.yaml file with environment.

values-dev.yaml
values-sit.yaml
values-prod.yaml

the only differences are some varialbes, such as environment (dev/sit/prod), and namespaces.

so when you run the deployment, it will be:

env=${ENVIRONMENT}
helm install -f values-${env}.yaml myredis ./redis
-- BMW
Source: StackOverflow

2/18/2020

So my question is:

How are you triggering these builds? Manually? GitHub Trigger? HTTP Trigger using the REST API?

so you're almost there for the building/pushing part, you would need to use substitution variables https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values

If you would be triggering the builds manually, you would edit the build trigger and change the sub variable for what you want it to be. GitHub Trigger -- this is a little more complex as you might want to do releases or branches. HTTP Trigger, same as manual, in your request you change the sub variable.

So here's part of one of our repo build files, as you will see there are different sub. variables we use, sometimes we want to build the image AND deploy to cluster, other times we just want to build or deploy.

steps:
# pull docker image
- name: 'gcr.io/cloud-builders/docker'
  id: pull-docker-image
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    docker pull ${_TAG_DOCKER_IMAGE} || exit 0

# build docker image
- name: 'gcr.io/cloud-builders/docker'
  id: build-docker-image
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    if [[ "${_BUILD_IMAGE}" == "true" ]]; then
       docker build -t ${_DOCKER_IMAGE_TAG} --cache-from ${_TAG_DOCKER_IMAGE} .;
    else
       echo "skipping ... BUILD_IMAGE=${_BUILD_IMAGE}";
    fi

# push docker image
- name: 'gcr.io/cloud-builders/docker'
  id: push-docker-image
  entrypoint: 'bash'
  args:
  - '-c'
  - |
     if [[ "${_BUILD_IMAGE}" == "true" ]]; then
        docker push ${_DOCKER_IMAGE_TAG};
     else
        echo "skipping ... BUILD_IMAGE=${_BUILD_IMAGE}";
     fi

# tag docker image
- name: 'gcr.io/cloud-builders/gcloud'
  id: tag-docker-image
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    if [[ "${_BUILD_IMAGE}" == "true" ]]; then
       gcloud container images add-tag ${_DOCKER_IMAGE_TAG} ${_TAG_DOCKER_IMAGE} -q;
    else
       echo "skipping ... BUILD_IMAGE=${_BUILD_IMAGE}";
    fi


# update service image on environment
- name: 'gcr.io/cloud-builders/kubectl'
  id: update service deployment image
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    if [[ "${_UPDATE_CLUSTER}" == "true" ]]; then
       /builder/kubectl.bash set image deployment $REPO_NAME master=${_DOCKER_IMAGE_TAG} --namespace=${_DEFAULT_NAMESPACE};
    else
       echo "skipping ... UPDATE_CLUSTER=${_UPDATE_CLUSTER}";
    fi
  env:
  - 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
  - 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'


# subs are needed because of our different ENVs
# _DOCKER_IMAGE_TAG = ['gcr.io/$PROJECT_ID/$REPO_NAME:gcb-${_COMPANY_ENV}-$SHORT_SHA', 'other']
# _COMPANY_ENV = ['dev', 'staging', 'prod']
# _DEFAULT_NAMESPACE = ['default'] or ['custom1', 'custom2']
# _CLOUDSDK_CONTAINER_CLUSTER = ['dev', 'prod']
# _CLOUDSDK_COMPUTE_ZONE = ['us-central1-a']
# _BUILD_IMAGE = ['true', 'false']
# _UPDATE_CLUSTER = ['true', 'false']
substitutions:
    _DOCKER_IMAGE_TAG: $DOCKER_IMAGE_TAG
    _COMPANY_ENV: dev
    _DEFAULT_NAMESPACE: default
    _CLOUDSDK_CONTAINER_CLUSTER: dev
    _CLOUDSDK_COMPUTE_ZONE: us-central1-a
    _BUILD_IMAGE: 'true'
    _UPDATE_CLUSTER: 'true'
options:
    substitution_option: 'ALLOW_LOOSE'
    env:
    - _TAG_DOCKER_IMAGE=gcr.io/$PROJECT_ID/$REPO_NAME:${_COMPANY_ENV}-latest
    - DOCKER_IMAGE_TAG=gcr.io/$PROJECT_ID/$REPO_NAME:gcb-${_COMPANY_ENV}-$SHORT_SHA
tags:
- '${_COMPANY_ENV}'
- 'build-${_BUILD_IMAGE}'
- 'update-${_UPDATE_CLUSTER}'

we have two workflows --

  1. github trigger builds and deploys under the 'dev' environment.
  2. we trigger via REST API https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.builds/create (we replace the variables via the request.json) -- this method also works using the gcloud builds --substitutions CLI.

Hope that answers your question!

-- Lance Sandino
Source: StackOverflow