gitlab-runner on a kubernetes cluster error while creating mount source path '/usr/share/ca-certificates/mozilla'

11/30/2018

I'm trying to get gitlab-runner "run" on a kubernetes cluster, after following the official doc -> https://docs.gitlab.com/runner/install/kubernetes.html (using kubernetes executor) I'm getting an error once I deploy:

Error: failed to start container "gitlab-runner": Error response from daemon: error while creating mount source path '/usr/share/ca-certificates/mozilla': mkdir /usr/share/ca-certificates/mozilla: read-only file system

I'm using the examples in that web and can't figure out why isn't allowing to create that dir (As I understand the default user is root)

Here my config-map.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner
  namespace: gitlab
data:
  config.toml: |
    concurrent = 1
    [[runners]]
      name = "Kubernetes Runner"
      url = "URL"
      token = "TOKEN"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab"

and this is the deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab-runner
  template:
    metadata:
      labels:
        name: gitlab-runner
    spec:
      containers:
        - args:
            - run
          image: gitlab/gitlab-runner:alpine-v11.5.0
          imagePullPolicy: Always
          name: gitlab-runner
          volumeMounts:
            - mountPath: /etc/gitlab-runner
              name: config
            - mountPath: /etc/ssl/certs
              name: cacerts
              readOnly: true
      restartPolicy: Always
      volumes:
        - configMap:
            name: gitlab-runner
          name: config
        - hostPath:
            path: /usr/share/ca-certificates/mozilla
          name: cacerts

Here is the complete list of events initializing the pod:

Events:
  Type     Reason                 Age                From                                                          Message
  ----     ------                 ----               ----                                                          -------
  Normal   Scheduled              29s                default-scheduler                                             Successfully assigned gitlab-runner-5b689c7cbc-hw6r5 to gke-my-project-dev-default-pool-0d32b263-6skk
  Normal   SuccessfulMountVolume  29s                kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  MountVolume.SetUp succeeded for volume "cacerts"
  Normal   SuccessfulMountVolume  29s                kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  MountVolume.SetUp succeeded for volume "config"
  Normal   SuccessfulMountVolume  29s                kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  MountVolume.SetUp succeeded for volume "default-token-6hr2h"
  Normal   Pulling                23s (x2 over 28s)  kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  pulling image "gitlab/gitlab-runner:alpine-v11.5.0"
  Normal   Pulled                 19s (x2 over 24s)  kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  Successfully pulled image "gitlab/gitlab-runner:alpine-v11.5.0"
  Normal   Created                19s (x2 over 24s)  kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  Created container
  Warning  Failed                 19s (x2 over 24s)  kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  Error: failed to start container "gitlab-runner": Error response from daemon: error while creating mount source path '/usr/share/ca-certificates/mozilla': mkdir /usr/share/ca-certificates/mozilla: read-only file system
  Warning  BackOff                14s                kubelet, gke-my-project-dev-default-pool-0d32b263-6skk  Back-off restarting failed container

Any clue will be appreciated

Thanks

-- ezekkiel
gitlab-ci-runner
google-kubernetes-engine
kubernetes

2 Answers

11/30/2018

From the logs, i am guessing you are using GKE. Google security mount your / file-system(see here). That's why you are getting error.

Try it by enabling privileged mode of the container:

containers:
   securityContext:
     privileged: true

If that does not work, then change /usr/share/ca-certificates/mozilla to /var/SOMETHING (not sure, this is good solution). If there are files in /usr/share/ca-certificates/mozilla, then move/copy them to /var/SOMETHING

-- nightfury1204
Source: StackOverflow

12/1/2018

Finally, I got it working here what I use to register and run the gitlab-runner on GKE

ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-cm
  namespace: gitlab
data:
  config.toml: |
    concurrent = 4
    check_interval = 30
  entrypoint: |
    #!/bin/bash

    set -xe
    cp /scripts/config.toml /etc/gitlab-runner/

    # Register the runner
    /entrypoint register --non-interactive \
      --url $GITLAB_URL \
      --tag-list "kubernetes, my_project" \
      --kubernetes-image "alpine:latest" \
      --kubernetes-namespace "gitlab" \
      --executor kubernetes \
      --config "/etc/gitlab-runner/config.toml" \
      --locked=false \
      --run-untagged=true \
      --description "My Project - Kubernetes Runner" \
      --kubernetes-privileged

    # Start the runner
    /entrypoint run --user=gitlab-runner \
      --working-directory=/home/gitlab-runner \
      --config "/etc/gitlab-runner/config.toml"

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitlab-runner
  template:
    metadata:
      labels:
        app: gitlab-runner
    spec:
      containers:
        - name: gitlab-runner
          image: gitlab/gitlab-runner:latest
          command: ["/bin/bash", "/scripts/entrypoint"]
          env:
            - name: GITLAB_URL
              value: "URL"
            - name: REGISTRATION_TOKEN
              value: "TOKEN"
            - name: KUBERNETES_NAMESPACE
              value: gitlab
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /var/secrets/google/key.json
          imagePullPolicy: Always
          volumeMounts:
            - name: config
              mountPath: /scripts
            - name: google-cloud-key
              mountPath: /var/secrets/google
      restartPolicy: Always
      volumes:
        - name: config
          configMap:
            name: gitlab-runner-cm
        - name: google-cloud-key
          secret:
            secretName: gitlab-runner-sa

And Autoscaling:

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: gitlab-runner-hpa
  namespace: gitlab
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gitlab-runner
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 50

I hope this helps someone trying to run a Gitlab Runner in a Kubernetes Cluster on Google Kubernetes Engine

-- ezekkiel
Source: StackOverflow