K
Q

GitLab CI runner can't connect to unix:///var/run/docker.sock in kubernetes

3/17/2017

GitLab's running in kubernetes cluster. Runner can't build docker image with build artifacts. I've already tried several approaches to fix this, but no luck. Here are some configs snippets:

.gitlab-ci.yml

image: docker:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay

stages:
  - build
  - package
  - deploy

maven-build:
  image: maven:3-jdk-8
  stage: build
  script: "mvn package -B --settings settings.xml"
  artifacts:
    paths:
      - target/*.jar

docker-build:
  stage: package
  script:
  - docker build -t gitlab.my.com/group/app .
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
  - docker push gitlab.my.com/group/app

config.toml

concurrent = 1
check_interval = 0

[[runners]]
  name = "app"
  url = "https://gitlab.my.com/ci"
  token = "xxxxxxxx"
  executor = "kubernetes"
  [runners.kubernetes]
    privileged = true
    disable_cache = true

Package stage log:

running with gitlab-ci-multi-runner 1.11.1 (a67a225)
  on app runner (6265c5)
Using Kubernetes namespace: default
Using Kubernetes executor with image docker:latest ...
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Waiting for pod default/runner-6265c5-project-4-concurrent-0h9lg9 to be running, status is Pending
Running on runner-6265c5-project-4-concurrent-0h9lg9 via gitlab-runner-3748496643-k31tf...
Cloning repository...
Cloning into '/group/app'...
Checking out 10d5a680 as master...
Skipping Git submodules setup
Downloading artifacts for maven-build (61)...
Downloading artifacts from coordinator... ok        id=61 responseStatus=200 OK token=ciihgfd3W
$ docker build -t gitlab.my.com/group/app .
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1

What am I doing wrong?

-- cardinal-gray
docker
gitlab
gitlab-ci-runner
kubernetes

Similar Questions

Define custom SSH key or find autogenerated SSH key in KOPs
Kubernetes events for container activity on node
Helm command with kubeconfig inline
Is there the concept of uploading a Deployment without causing pods to start?

4 Answers

10/25/2019

I had the same problem, and I could not get the above workarounds to work for me (I did not try the volumes trick mentioned by @fkpwolf).

Now GitLab has an alternative solution by using Kaniko, which did work for me:

The .gitlab-ci.yaml could then be something like this, in that case:

stages:
  - build
  - package
  - deploy

maven-build:
  image: maven:3-jdk-8
  stage: build
  script: "mvn package -B --settings settings.xml"
  artifacts:
    paths:
      - target/*.jar

docker-kaniko-build:
  stage: package
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"gitlab.my.com\":{\"username\":\"gitlab-ci-token\",\"password\":\"$CI_BUILD_TOKEN\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination gitlab.my.com/group/app

From the GitLab docs it is mentioned that:

kaniko solves two problems with using the docker-in-docker build method:

  • Docker-in-docker requires privileged mode in order to function, which is a significant security concern.
  • Docker-in-docker generally incurs a performance penalty and can be quite slow.

See: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html

-- audunsol
Source: StackOverflow
3/20/2017

Don't need to use this:

DOCKER_DRIVER: overlay

cause it seems like OVERLAY isn't supported, so svc-0 container is unable to start with it:

$ kubectl logs -f `kubectl get pod |awk '/^runner/{print $1}'` -c svc-0
time="2017-03-20T11:19:01.954769661Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2017-03-20T11:19:01.955720778Z" level=info msg="libcontainerd: new containerd process, pid: 20"
time="2017-03-20T11:19:02.958659668Z" level=error msg="'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded."

Also, add export DOCKER_HOST="tcp://localhost:2375" to the docker-build:

 docker-build:
  stage: package
  script:
  - export DOCKER_HOST="tcp://localhost:2375"
  - docker build -t gitlab.my.com/group/app .
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab.my.com/group/app
  - docker push gitlab.my.com/group/app
-- cardinal-gray
Source: StackOverflow
3/18/2017

When using Kubernetes, you have to adjust your Build image to connect with the Docker engine.

Add to your build image:

DOCKER_HOST=tcp://localhost:2375

Quote from the docs:

Running the docker:dind also known as the docker-in-docker image is also possible but sadly needs the containers to be run in privileged mode. If you're willing to take that risk other problems will arise that might not seem as straight forward at first glance. Because the docker daemon is started as a service usually in your .gitlab-ci.yaml it will be run as a separate container in your pod. Basically containers in pods only share volumes assigned to them and an IP address by wich they can reach each other using localhost. /var/run/docker.sock is not shared by the docker:dind container and the docker binary tries to use it by default. To overwrite this and make the client use tcp to contact the docker daemon in the other container be sure to include DOCKER_HOST=tcp://localhost:2375 in your environment variables of the build container.

Gitlab-CI on Kubernetes

-- opHASnoNAME
Source: StackOverflow
9/29/2019

based on @Yarik 's comment what worked for me was

- export DOCKER_HOST=$DOCKER_PORT

no other answers worked.

-- Nithin
Source: StackOverflow