Best practices when implementing CI/CD pipeline using GitHub/Jenkins/Kubernetes
This question is more advice related so I hope its not flagged for anything. Just really need help :(
Trying to implement CI/CD using GitHub/Jenkins/Kubernetes.
On a highlevel this is what should happen:
- Build on Jenkins
- Push to container registry
- Deploy built image on Kubernetes development cluster
- Once testing finished on Development cluster, deploy it on a client testing cluster and finally production cluster
So far this is what I have created a job on Jenkins which will be triggered using a Github hook. This job is responsible for the following things:
- Checkout from GitHub
- Run unit tests / call REST API and send unit test results
- Build artifacts using maven / call REST API and inform if build success or fail
- Build docker image
- Push docker image to container registry (docker image will have incremented versions which match with the BUILD_NUMBER environment variable)
The above stated tasks are more or less completed and I dont need much assitance with it (unless anyone thinks the aforementioned steps are not best practice)
I do need help with the part where I deploy to the Kubernetes cluster.
For local testing, I have set up a local cluster using Vagrant boxes and it works. In order to deploy the built image on the development cluster, I am thinking about doing it like this: Point Jenkins build server to Kubernetes development cluster Deploy using deployment.yml and service.yml (available in my repo) This part I need help with...
Is this wrong practice? Is there a better/easier way to do it?
Also is there a way to migrate between clusters? Ex: Development cluster to client testing cluster and client testing cluster to production cluster etc
When searching on the internet, the name Helm comes up a lot but I am not sure if it will be applicable to my use case. I would test it and see but I am a bit hard pressed for time which is why I cant
Would appreciate any help y'all could provide.
Thanks a lot
Similar Questions
3 Answers
There are countless ways of doing this. Take Helm out for now as you are just starting.
If you are already using Github and docker , then I would just recommend you to push your code/changes/config/Dockerfile to Github that will auto trigger a docker build on Dockerhub ( maybe jenkins in ur case if u dont want to use dockerhub for builds ) , it can be a multi-stage docker build where you can build code , run tests , throw away dev environmenet , and finally produce a producion docker image , once the image is produced , it will triger a web hook to your kubernetes deployment job/manifests to deploy on to test evironmenet , followed by manual triiger to deploy to production.
The docker images can be tagged based on SHA of the commits in Github/Git so that you can deploy and rollback based on commits.
Reference: https://cloud.google.com/kubernetes-engine/docs/tutorials/gitops-cloud-build
Here is my Gitlab implementation of Gtips workflow:
# Author , IjazAhmad
image: docker:latest
stages:
- build
- test
- deploy
services:
- docker:dind
variables:
CI_REGISTRY: dockerhub.example.com
CI_REGISTRY_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH
DOCKER_DRIVER: overlay2
before_script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
docker-build:
stage: build
script:
- docker pull $CI_REGISTRY_IMAGE:latest || true
- docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .
docker-push:
stage: build
script:
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
unit-tests:
stage: test
script:
- echo "running unit testson the image"
- echo "running security testing on the image"
- echo "pushing the results to build/test pipeline dashboard"
sast:
stage: test
script:
- echo "running security testing on the image"
- echo "pushing the results to build/test pipeline dashboard"
dast:
stage: test
script:
- echo "running security testing on the image"
- echo "pushing the results to build/test pipeline dashboard"
testing:
stage: deploy
script:
- sed -i "s|CI_IMAGE|$CI_REGISTRY_IMAGE|g" k8s-configs/deployment.yaml
- sed -i "s|TAG|$CI_COMMIT_SHA|g" k8s-configs/deployment.yaml
- kubectl apply --namespace webproduction-test -f k8s-configs/
environment:
name: testing
url: https://testing.example.com
only:
- branches
staging:
stage: deploy
script:
- sed -i "s|CI_IMAGE|$CI_REGISTRY_IMAGE|g" k8s-configs/deployment.yaml
- sed -i "s|TAG|$CI_COMMIT_SHA|g" k8s-configs/deployment.yaml
- kubectl apply --namespace webproduction-stage -f k8s-configs/
environment:
name: staging
url: https://staging.example.com
only:
- master
production:
stage: deploy
script:
- sed -i "s|CI_IMAGE|$CI_REGISTRY_IMAGE|g" k8s-configs/deployment.yaml
- sed -i "s|TAG|$CI_COMMIT_SHA|g" k8s-configs/deployment.yaml
- kubectl apply --namespace webproduction-prod -f k8s-configs/
environment:
name: production
url: https://production.example.com
when: manual
only:
- master
Links:
Trigger Jenkins builds by pushing to Github
I understand that you are trying to implement GitOps, my advice is to review this article where you can start to figure out a little bit more about the components you need.
https://www.weave.works/blog/managing-helm-releases-the-gitops-way
Basically, you need to implement your own helm charts for your custom services and manage it using flux, I recommend to use a different repository per environment and leave flux to manage the deployment to each environment based on the state of the master branch on the repo.
Look at spinnaker for continuous delivery. After the image is built and pushed to registry, have a web hook in spinnaker trigger a deployment to required kubernetes cluster. Spinnaker works well with kubernetes and you definitely should try it out