I would like to ask you for a help/advice with the following issue. We are using Bamboo as our CI and we have remote bamboo agents running on k8s.
In our build we have step that creates a Docker image when tests ran correctly. To remote bamboo agents we are exposing Docker via docker.socket
. When we had only one remote bamboo agent (to test how it works) everything was working correctly but recently we have increased the number of remote agents. Now it happen quite oft that a build gets stuck in docker image build step and will not move. We have to stop the build and run it again. Usually in logs is no useful info, but once in while this will appear.
24-May-2017 16:04:54 Execution failed for task ':...'.
24-May-2017 16:04:54 > Docker execution failed
24-May-2017 16:04:54 Command line [docker build -t ...] returned:
24-May-2017 16:04:54 time="2017-05-24T16:04:54+02:00" level=info msg="device or resource busy"
This how our k8s deployment
looks like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: bamboo-agent
namespace: backend-ci
spec:
replicas: 5
template:
metadata:
labels:
app: bamboo-agent
spec:
containers:
- name: bamboo-agent
stdin: true
resources:
.
env:
.
.
.
ports:
- .
volumeMounts:
- name: dockersocket
mountPath: /var/run/docker.sock
volumes:
- hostPath:
path: /var/run/docker.sock
name: dockersocket
And here is Dockerfile
for remote bamboo agent.
FROM java:8
ENV CI true
RUN apt-get update && apt-get install -yq curl && apt-get -yqq install docker.io && apt-get install tzdata -yq
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl && mv kubectl /usr/local/bin
RUN echo $TZ | tee /etc/timezone
RUN dpkg-reconfigure --frontend noninteractive tzdata
RUN wget -O /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64
RUN chmod +x /usr/local/bin/dumb-init
ADD run.sh /root
ADD .dockercfg /root
ADD config /root/.kube/
ADD config.json /root/.docker/
ADD gradle.properties /root/.gradle/
ADD bamboo-capabilities.properties /root
ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]
CMD /root/run.sh
Is there some way how to solve this issue? And is exposing docker.socket
a good solution or is there some better approach?
I have read few articles about Docker in docker but I do not like --privileged
mode.
If you need some other information I will try to provide them.
Thank you.
One of the things you can do is run your builds on rkt while running kubernetes on docker?