Continuous deployment to Google Kubernetes Engine with google container registry

6/4/2020

I try to build a continuous deployment pipeline for my GKE cluster. I use my own gitlab-runner as CI pipeline build and push images to gcr.io/PROJECT/APP:google tag there.

Is there any possibility to implement the rolling restart of the containers that use this image after its update? I have seen a lot of examples of how to do it using Jenkins and Google Source Repository directly in a Kubernetes cluster, but is there any possibility to trigger only on image changes?

I have found something that I need here https://cloud.google.com/container-registry/docs/configuring-notifications. But still, I have no idea how to connect these notifications with the cluster.

-- Oleg
continuous-deployment
google-kubernetes-engine
kubernetes

1 Answer

6/19/2020

After some tests, finally I made it works using PubSub and Kubernetes Cronjob.

How it works:

When a new image is pushe to Container Registry, a message is sent to Pub/Sub that contains some important data, like this:

{
  "action":"INSERT",
  "digest":"gcr.io/my-project/hello-world@sha256:6ec128e26cd5...",
  "tag":"gcr.io/my-project/hello-world:1.1"
}

The action with value INSERT means that a new image was pushed to Pub/Sub. The key tag contains the name of the image that was pushed.

So, all we need to do is, read this data and update the deployment image.

First, We need some code to retrieve the message from Pub/Sub. Unfortunately I can't find anything "ready" for this task, so you need to create you own. Here there are some examples of how you can retrive the messages from Pub/Sub.

As a proof-of-concept, my choice was to use a shell script (imageUpdater.sh) to retrieve the message from Pub/Sub and execute the kubectl set image... command to update the deployment image.

Second, create a Cronjob using the first code to read the message and update the deployment.

In my example, I've create a docker image with gcloud and kubectl command to perform the tasks, you can find the code Here.

But, to make it all work, your must to grant permissions to job pod's to execute "kubectl set image", and for this wee need to configure the RBAC permissions.

To create a "isolated" PoC, I will create every resources describe here in a new namespace named "myns". The RBAC will only work in this name space, because Role is namespace, if you what to use in all namespaces, change to ClusterRole

1. PubSub Configuration

First of all, you need to configure Container Registry to sent messages to Pub/Sub. You can follow this guide.

In this example I will use a nginx image to demonstrate.

gcloud pubsub topics create projects/[PROJECT-ID]/topics/gcr

From the system where Docker images are pushed or tagged run the following command:

gcloud pubsub subscriptions create nginx --topic=gcr

2. GKE Cluster

GKE needs permission to access Pub/Sub, and it can be done only when a new cluster is created using the --scope "https://www.googleapis.com/auth/pubsub". So I will create a new cluster to our example:

Create a new cluster to our example:

gcloud container clusters create "my-cluster" --num-nodes "1" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/pubsub","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append"

More information about scopes here.

Getting the cluster credentials:

gcloud container clusters get-credentials my-cluster

3. Configuring RBAC permissions

As mentioned before all resources will be created in the namespace myns. So let's create the namespace:

kubectl create ns myns

After that we can create a new Service account called sa-image-update and apply the RBAC permissions:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-image-update
  namespace: myns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: myns
  name: role-set-image
rules:
- apiGroups: ["apps", "extensions"]
  resources: ["deployments"]
  verbs: ["get", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rolebinding-set-image
  namespace: myns
roleRef:
  kind: Role
  name: role-set-image
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: sa-image-update
  namespace: myns

4. Confimap

To make it easiest as possible, I will create a configmap with the shell script file that will be mounted and executed by the pod:

# Download script
wget https://raw.githubusercontent.com/MrKoopaKiller/docker-gcloud-kubectl/master/imageUpdater.sh
# Create configmap
kubectl create configmap imageupdater -n myns --from-file imageUpdater.sh

5. CronJob

The shell script need 3 variables to work:

PROJECT-NAME: The name of gcloud project DEPLOYMENT-NAME: The name of the deployment that will be updated IMAGE-NAME: The name of the image to update without the tag.

In this case, my example deployment will be called nginx and the image nginx.

The image is from the dockerfile I mentioned before, you can find here and build it.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: image-updater
  namespace: myns
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: sa-image-update
          volumes:
            - name: imageupdater
              configMap:
                name: imageupdater
          containers:
          - name: image-updater
            image: <your_custom_image>
            volumeMounts:
            - name: imageupdater
              mountPath: /bin/imageUpdater.sh
              subPath: imageUpdater.sh
            command: ['bash', '/bin/imageUpdater.sh', 'PROJECT-NAME', 'nginx', 'nginx']
          restartPolicy: Never

OK, everything is done. Now we need to create a deployment as example to demonstrated:

Example deployment: nginx

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: myns
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - name: http
          containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: myns
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Ok, now when your gitlab push a new image to COntainer Registry, a message will be sent to Pub/Sub, the cronjob will runs every 2 minutes, verify if the image name is nginx and if yes, will run the kubectl set image.

References:

https://medium.com/better-programming/k8s-tips-using-a-serviceaccount-801c433d0023

https://cloud.google.com/solutions/integrating-microservices-with-pubsub#creating_a_gke_cluster

-- Mr.KoopaKiller
Source: StackOverflow