K
Q

Pending status of pods when running Gitlab CI runner on Kubernetes

10/16/2017

I am currently trying to use a Kubernetes cluster for the Gitlab CI. While following the not so good docs (https://docs.gitlab.com/runner/install/kubernetes.html), what I did was manually register a runner with the token from Gitlab CI section so I could get another token and use it in the ConfigMap I use for the deployment.

-ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner
  namespace: gitlab
data:
  config.toml: |
    concurrent = 4
    [[runners]]
        name = "Kubernetes Runner"
        url = "https://url/ci"
        token = "TOKEN"
        executor = "kubernetes"
        [runners.kubernetes]
            namespace = "gitlab"

-Deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab
spec:
  replicas: 4
  selector:
    matchLabels:
      name: gitlab-runner
  template:
    metadata:
      labels:
        name: gitlab-runner
    spec:
      containers:
      - args:
        - run
        image: gitlab/gitlab-runner:latest
        imagePullPolicy: Always
        name: gitlab-runner
        volumeMounts:
        - mountPath: /etc/gitlab-runner
          name: config
      restartPolicy: Always
      volumes:
      - configMap:
          name: gitlab-runner
        name: config

With these two I get to see the runner in the Gitlab Runner section but whenever I start a job, the new created pods stay in pending status.

I would like to fix it but all I know is that the nodes and pods get these events:

-Pods:

Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                    -------------   --------        ------                  -------
  35s           4s              7       {default-scheduler }                    Warning         FailedScheduling        No nodes are available that match all of the following predicates:: MatchNodeSelector (2).

-Nodes:

Events:
  FirstSeen     LastSeen        Count   From                                            SubObjectPath   Type            Reason                                  Message
  ---------     --------        -----   ----                                            -------------   --------        ------                                  -------
  4d            31s             6887    {kubelet gitlab-ci-hc6k3ffax54o-master-0}                       Warning         FailedNodeAllocatableEnforcement        Failed to update Node Allocatable Limits "": failed to set supported cgroup subsystems for cgroup : Failed to set config for supported subsystems : failed to write 3783761920 to memory.limit_in_bytes: write /rootfs/sys/fs/cgroup/memory/memory.limit_in_bytes: invalid argument 

Any idea of why this is happening?

EDIT: kubectl describe added:

Name:           runner-45384765-project-1570-concurrent-00mb7r
Namespace:      gitlab
Node:           /
Labels:         <none>
Status:         Pending
IP:
Controllers:    <none>
Containers:
  build:
    Image:      blablabla:latest
    Port:
    Command:
      sh
      -c
      if [ -x /usr/local/bin/bash ]; then
        exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
        exec /usr/bin/bash
elif [ -x /bin/bash ]; then
        exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
        exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
        exec /usr/bin/sh
elif [ -x /bin/sh ]; then
        exec /bin/sh
else
        echo shell not found
        exit 1
fi


    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-1qm5n (ro)
      /vcs from repo (rw)
    Environment Variables:
      CI_PROJECT_DIR:           blablabla
      CI_SERVER:                yes
      CI_SERVER_TLS_CA_FILE:    -----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla 
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----

      CI:                       true
      GITLAB_CI:                true
      CI_SERVER_NAME:           GitLab
      CI_SERVER_VERSION:        9.5.5-ee
      CI_SERVER_REVISION:       cfe2d5c
      CI_JOB_ID:                5625
      CI_JOB_NAME:              pylint
      CI_JOB_STAGE:             build
      CI_COMMIT_SHA:            ece31293f8eeb3a36a8585b79d4d21e0ebe8008f
      CI_COMMIT_REF_NAME:       master
      CI_COMMIT_REF_SLUG:       master
      CI_REGISTRY_USER:         gitlab-ci-token
      CI_BUILD_ID:              5625
      CI_BUILD_REF:             ece31293f8eeb3a36a8585b79d4d21e0ebe8008f
      CI_BUILD_BEFORE_SHA:      ece31293f8eeb3a36a8585b79d4d21e0ebe8008f
      CI_BUILD_REF_NAME:        master
      CI_BUILD_REF_SLUG:        master
      CI_BUILD_NAME:            pylint
      CI_BUILD_STAGE:           build
      CI_PROJECT_ID:            1570
      CI_PROJECT_NAME:          blablabla
      CI_PROJECT_PATH:          blablabla
      CI_PROJECT_PATH_SLUG:     blablabla
      CI_PROJECT_NAMESPACE:     vcs
      CI_PROJECT_URL:           https://blablabla
      CI_PIPELINE_ID:           2574
      CI_CONFIG_PATH:           .gitlab-ci.yml
      CI_PIPELINE_SOURCE:       push
      CI_RUNNER_ID:             111
      CI_RUNNER_DESCRIPTION:    testing on kubernetes
      CI_RUNNER_TAGS:           docker-image-build
      CI_REGISTRY:              blablabla
      CI_REGISTRY_IMAGE:        blablabla
      PYLINTHOME:               ./pylint-home
      GITLAB_USER_ID:           2277
      GITLAB_USER_EMAIL:        blablabla
  helper:
    Image:      gitlab/gitlab-runner-helper:x86_64-a9a76a50
    Port:
    Command:
      sh
      -c
      if [ -x /usr/local/bin/bash ]; then
        exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
        exec /usr/bin/bash
elif [ -x /bin/bash ]; then
        exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
        exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
        exec /usr/bin/sh
elif [ -x /bin/sh ]; then
        exec /bin/sh
else
        echo shell not found
        exit 1
fi


    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-1qm5n (ro)
      /vcs from repo (rw)
    Environment Variables:
      CI_PROJECT_DIR:           blablabla
      CI_SERVER:                yes
      CI_SERVER_TLS_CA_FILE:    -----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
blablabla
-----END CERTIFICATE-----

      CI:                       true
      GITLAB_CI:                true
      CI_SERVER_NAME:           GitLab
      CI_SERVER_VERSION:        9.5.5-ee
      CI_SERVER_REVISION:       cfe2d5c
      CI_JOB_ID:                5625
      CI_JOB_NAME:              pylint
      CI_JOB_STAGE:             build
      CI_COMMIT_SHA:            ece31293f8eeb3a36a8585b79d4d21e0ebe8008f
      CI_COMMIT_REF_NAME:       master
      CI_COMMIT_REF_SLUG:       master
      CI_REGISTRY_USER:         gitlab-ci-token
      CI_BUILD_ID:              5625
      CI_BUILD_REF:             ece31293f8eeb3a36a8585b79d4d21e0ebe8008f
      CI_BUILD_BEFORE_SHA:      ece31293f8eeb3a36a8585b79d4d21e0ebe8008f
      CI_BUILD_REF_NAME:        master
      CI_BUILD_REF_SLUG:        master
      CI_BUILD_NAME:            pylint
      CI_BUILD_STAGE:           build
      CI_PROJECT_ID:            1570
      CI_PROJECT_NAME:          blablabla
      CI_PROJECT_PATH:          blablabla
      CI_PROJECT_PATH_SLUG:     blablabla
      CI_PROJECT_NAMESPACE:     vcs
      CI_PROJECT_URL:           blablabla
      CI_PIPELINE_ID:           2574
      CI_CONFIG_PATH:           .gitlab-ci.yml
      CI_PIPELINE_SOURCE:       push
      CI_RUNNER_ID:             111
      CI_RUNNER_DESCRIPTION:    testing on kubernetes
      CI_RUNNER_TAGS:           docker-image-build
      CI_REGISTRY:              blablabla
      CI_REGISTRY_IMAGE:        blablabla
      PYLINTHOME:               ./pylint-home
      GITLAB_USER_ID:           2277
      GITLAB_USER_EMAIL:        blablabla
Conditions:
  Type          Status
  PodScheduled  False
Volumes:
  repo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-1qm5n:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-1qm5n
QoS Class:      BestEffort
Tolerations:    <none>
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                    -------------   --------        ------                  -------
  39s           8s              7       {default-scheduler }                    Warning         FailedScheduling        No nodes are available that match all of the following predicates:: MatchNodeSelector (2).
-- djuarez
gitlab
gitlab-ci
gitlab-ci-runner
kubectl
kubernetes

Similar Questions

How to create secrets using Kubernetes Python client?
Why does Kubernetes showing the nodes as ready even if they are not reachable?
Is there any community hosted distributed cloud solution?
Is there a way to use a higher version kubectl to request a lower version kubernetes?

2 Answers

10/26/2017

Because of:

selector:
    matchLabels:
      name: gitlab-runner

No pod is able to retrieve a job with that label.

Removing the selector would be enough, there is no need for it.

-- djuarez
Source: StackOverflow
5/29/2018

@djuarez As long the deployment selector matches the pods label in the template section, and in this case from what I can see that is the case:

selector:
  matchLabels:
    name: gitlab-runner
template:
  metadata:
    labels:
      name: gitlab-runner

it should not be a problem; provided the correct API is used which in this case apiVersion: extensions/v1beta1 is also correct. The describe output shows MatchNodeSelector which has nothing to do with the Deployment selector. My guess is the full Deployment config is not being shown here and something else is wrong like trying to deploy the pods to specific nodes via nodeSeletor that do not have the requested label in the nodeSelector condition.

-- IgorC
Source: StackOverflow