Because of increasing build time of our pipeline, we have tried several things to improve it. One step which was taking quite some time was the docker images push step which was running sequentially. Being 12 images, this step was taking 12-14 minutes and we decided trying to push the images in parallel (under the considerate that this will take the time from 12-14 to 2-4 minutes).
Tried multiple steps under a publish images stage, but it fails.
- name: Publish images
steps:
- publishImageConfig:
dockerfilePath: ./frontend/deployment/Dockerfile
buildContext: ./frontend
tag: registry.remote.com/remote/frontend-${CICD_EXECUTION_ID}
pushRemote: true
registry: registry.remote.com
- publishImageConfig:
dockerfilePath: ./gateway/backend/src/Dockerfile
buildContext: ./gateway/backend
tag: registry.remote.com/remote/backend-${CICD_EXECUTION_ID}
pushRemote: true
registry: registry.remote.com
[...]
One image is pushed, but all the rest fail with Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I've also tried increasing the --max-concurrent-uploads from /etc/docker/daemon.json without any success.
Docker's /var/lib/docker can only be managed by a single daemon. If you want to publish more than one there is a workaround for that. Try something like this:
stages:
- name: Publish images_1
steps:
- publishImageConfig:
dockerfilePath: ./frontend/deployment/Dockerfile
buildContext: ./frontend
tag: registry.remote.com/remote/frontend-${CICD_EXECUTION_ID}
pushRemote: true
registry: registry.remote.com
- name: Publish images_2
steps:
- publishImageConfig:
dockerfilePath: ./gateway/backend/src/Dockerfile
buildContext: ./gateway/backend
tag: registry.remote.com/remote/backend-${CICD_EXECUTION_ID}
pushRemote: true
registry: registry.remote.com
env:
PLUGIN_STORAGE_PATH: /var/lib/docker_2
[...]
This bug was already reported in this thread, you can find more info there. Issue was supposed to be fixed in Rancher v2.2 but some people still experience this in v2.3. However, the workaround is still valid.
I am posting this answer as a community wiki because the fix was not my original idea.
Please let me know if that helped.