I try to build a continuous deployment pipeline for my GKE cluster. I use my own gitlab-runner as CI pipeline build and push images to gcr.io/PROJECT/APP:google tag there.
Is there any possibility to implement the rolling restart of the containers that use this image after its update? I have seen a lot of examples of how to do it using Jenkins and Google Source Repository directly in a Kubernetes cluster, but is there any possibility to trigger only on image changes?
I have found something that I need here https://cloud.google.com/container-registry/docs/configuring-notifications. But still, I have no idea how to connect these notifications with the cluster.
After some tests, finally I made it works using PubSub and Kubernetes Cronjob.
When a new image is pushe to Container Registry, a message is sent to Pub/Sub that contains some important data, like this:
{
"action":"INSERT",
"digest":"gcr.io/my-project/hello-world@sha256:6ec128e26cd5...",
"tag":"gcr.io/my-project/hello-world:1.1"
}
The action
with value INSERT
means that a new image was pushed to Pub/Sub.
The key tag
contains the name of the image that was pushed.
So, all we need to do is, read this data and update the deployment image.
First, We need some code to retrieve the message from Pub/Sub. Unfortunately I can't find anything "ready" for this task, so you need to create you own. Here there are some examples of how you can retrive the messages from Pub/Sub.
As a proof-of-concept, my choice was to use a shell script (imageUpdater.sh) to retrieve the message from Pub/Sub and execute the
kubectl set image...
command to update the deployment image.
Second, create a Cronjob using the first code to read the message and update the deployment.
In my example, I've create a docker image with gcloud and kubectl command to perform the tasks, you can find the code Here.
But, to make it all work, your must to grant permissions to job pod's to execute "kubectl set image", and for this wee need to configure the RBAC permissions.
To create a "isolated" PoC, I will create every resources describe here in a new namespace named "myns". The RBAC will only work in this name space, because
Role
is namespace, if you what to use in all namespaces, change toClusterRole
First of all, you need to configure Container Registry to sent messages to Pub/Sub. You can follow this guide.
In this example I will use a nginx image to demonstrate.
gcloud pubsub topics create projects/[PROJECT-ID]/topics/gcr
From the system where Docker images are pushed or tagged run the following command:
gcloud pubsub subscriptions create nginx --topic=gcr
GKE needs permission to access Pub/Sub, and it can be done only when a new cluster is created using the --scope "https://www.googleapis.com/auth/pubsub"
. So I will create a new cluster to our example:
Create a new cluster to our example:
gcloud container clusters create "my-cluster" --num-nodes "1" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/pubsub","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append"
More information about scopes
here.
Getting the cluster credentials:
gcloud container clusters get-credentials my-cluster
As mentioned before all resources will be created in the namespace myns
. So let's create the namespace:
kubectl create ns myns
After that we can create a new Service account called sa-image-update
and apply the RBAC permissions:
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-image-update
namespace: myns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: myns
name: role-set-image
rules:
- apiGroups: ["apps", "extensions"]
resources: ["deployments"]
verbs: ["get", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rolebinding-set-image
namespace: myns
roleRef:
kind: Role
name: role-set-image
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: sa-image-update
namespace: myns
To make it easiest as possible, I will create a configmap with the shell script file that will be mounted and executed by the pod:
# Download script
wget https://raw.githubusercontent.com/MrKoopaKiller/docker-gcloud-kubectl/master/imageUpdater.sh
# Create configmap
kubectl create configmap imageupdater -n myns --from-file imageUpdater.sh
The shell script need 3 variables to work:
PROJECT-NAME
: The name of gcloud project
DEPLOYMENT-NAME
: The name of the deployment that will be updated
IMAGE-NAME
: The name of the image to update without the tag.
In this case, my example deployment will be called nginx
and the image nginx
.
The image is from the dockerfile I mentioned before, you can find here and build it.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: image-updater
namespace: myns
spec:
schedule: "*/2 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: sa-image-update
volumes:
- name: imageupdater
configMap:
name: imageupdater
containers:
- name: image-updater
image: <your_custom_image>
volumeMounts:
- name: imageupdater
mountPath: /bin/imageUpdater.sh
subPath: imageUpdater.sh
command: ['bash', '/bin/imageUpdater.sh', 'PROJECT-NAME', 'nginx', 'nginx']
restartPolicy: Never
OK, everything is done. Now we need to create a deployment as example to demonstrated:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: myns
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- name: http
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
namespace: myns
spec:
type: LoadBalancer
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
Ok, now when your gitlab push a new image to COntainer Registry, a message will be sent to Pub/Sub, the cronjob will runs every 2 minutes, verify if the image name is nginx
and if yes, will run the kubectl set image
.
https://medium.com/better-programming/k8s-tips-using-a-serviceaccount-801c433d0023
https://cloud.google.com/solutions/integrating-microservices-with-pubsub#creating_a_gke_cluster