I use GitLab Runner on my Kubernetes cluster to run CI jobs. I want to make build jobs run faster.
To make them faster, I reuse Docker image from the previous build (tagged as latest). Build time has decreased, but now the bottleneck is the pull command which takes about 60-70% of the time.
Here's the snippet of the .gitlab-ci.yml:
build:sheets:
stage: build
image: docker:stable
services:
- docker:dind
before_script:
- echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" "$CI_REGISTRY" --password-stdin
script:
- docker pull $SHEETS_LATEST || true
- docker build --cache-from $SHEETS_LATEST --tag $SHEETS_TAG --tag $SHEETS_LATEST .
- docker push $SHEETS_TAG
- docker push $SHEETS_LATEST
I use Gitlab Registry and thus pull command requires a lot of communication between my cluster and the registry.
So I have a couple of questions here:
Is it worth effort to deploy my own docker registry on the cluster to save a couple of minutes per build?
Is there a way to save an image somewhere on cluster to not pull from registry every time?
Gitlab shared runners don't have your previous image available (as you noticed). So your options are either pull it like you are doing, or go with your own runners. Last time I looked, it wasn't available with the kubernetes runner that you can install on the gitlab kubernetes integration page, so I had to deploy my own runners.
Here is a deployment that creates 4 runners (todo: figure out autoscaling) and registers them as a postStart event. You'll need to get your registration token from your settings/cicd page for your project or group (https://gitlab.com/[user]/[project]/settings/ci_cd). And you'll have to disable shared runners from the same page.
kind: Deployment
metadata:
name: gitlabrunner
labels:
app: gitlabrunner
spec:
replicas: 4
selector:
matchLabels:
app: gitlabrunner
template:
metadata:
labels:
app: gitlabrunner
name: gitlabrunner
spec:
containers:
- name: gitlabrunner
image: gitlab/gitlab-runner
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "gitlab-runner register -n --url https://gitlab.com/ --registration-token [YOUR_REGISTRATION_TOKEN] --executor docker --description 'My Docker Runner' --docker-image 'docker:stable' --docker-volumes /var/run/docker.sock:/var/run/docker.sock"]
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sockYou can run dedicated pod with Docker daemon inside with persistent volume mounted to /var/lib/docker. After that you expose this pod using service named for example "docker". After that in your CI jobs you define environment variable "DOCKER_HOST=tcp://docker".
That way all your builds and images are stored in Docker daemon running in dedicated pod. You don't lose time pulling images and also you builds run faster because of Docker build cache.
Just remember to monitor disk space usage on persistent volume and trigger "docker system prune" periodically.
There are several methods to build Docker images in Pipeline quicker.
Here you've to mount host's /var/lib/docker.sock inside your build container. It is vulnerable. But still too fast.
Here you can use dind image which has both docker cli and daemon. It doesn't communicate to host's docker daemon. The entire build process will happen within the pod. Not 100% secure but faster.
These are Daemoneless next-generation image build tools and which does not depend on Docker daemon. More secure than DOOD and DIND also faster and supports caching.