K
Q

Losing my mind trying to get the VM to talk to Kubernetes in Google Cloud

10/14/2017

I've been pulling my hair out over this for too many hours... I'm pretty new to kubernetes so I know I must be missing something.

"ERROR: Job failed (system failure): the server does not allow access to the requested resource (post pods)"

We have a GitLab instance setup on a VM, and another VM with the GitLab runner installed. Both live in Google Cloud Compute Engine.

We also have a Kubernetes cluster spun up on Google Cloud.

When the runner attempts to run, it results in the following:

Running with gitlab-runner 10.0.2 (a9a76a50)
  on rd-002-optic-nexus (21590677)
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image docker:git ...
ERROR: Job failed (system failure): the server does not allow access to the requested resource (post pods)

Due to the Runner being "external" to the cluster, my only option is to authenticate to the API server via "client certificate" authentication.

I'm using the cluster ca.crt provided from the Google Cloud Console, and have followed Kubernetes guide to create a client cert. However, I just can't get it to work.

  • I can ping the Kubernetes server no problem from the (Runner) VM.
  • GitLab is hooked up to the runner and does attempt to use it (hence the output above).
  • In the kubernetes cluster, we've:
    • created the namespace "gitlab".
    • created a service account "gitlab-sa" in the "gitlab" namespace.
    • generated and approved the client certificate ok (csr details below).
  • The certificates on the runner have read permissions for everyone.

I must be missing something somewhere.

GitLab Runner Config

concurrent = 1
check_interval = 0
[[runners]]
  name = "rd-002-optic-nexus"
  url = "https://our.gitlab.instance.com/"
  token = "21590677f31b57bce610ef3f4cb20d"
  executor = "kubernetes"
  [runners.kubernetes]
    host = "https://111.222.x.xxx"
    cert_file = "/usr/local/share/ca-certificates/kube-client.crt"
    key_file = "/usr/local/share/ca-certificates/kube-client.key"
    ca_file = "/usr/local/share/ca-certificates/kubernetes-ca.crt"
    namespace = "gitlab"
    namespace_overwrite_allowed = ""
    privileged = false
    cpu_limit = "1"
    memory_limit = "1Gi"
    service_cpu_limit = "1"
    service_memory_limit = "1Gi"
    helper_cpu_limit = "500m"
    helper_memory_limit = "100Mi"
    [runners.kubernetes.node_selector]
      gitlab = "true"

Kubernetes Client CSR

apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: gitlab-sa.gitlab
spec:
  groups:
  - system:authenticated
  request: $(cat server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth

Any thoughts? Anything I'm missing?

-- Christopher
gitlab-ci-runner
google-kubernetes-engine
kubernetes

Similar Questions

Spring Cloud Task can not wok on SCDF Kubernetes
One large machine or cluster of small machines for Jmeter load generator in load testing?
Multiple Kubernetes pods sharing the same host-path/pvc will duplicate output
OpenShift and hostnetwork=true

2 Answers

10/15/2017

The user account provided must have permission to create, list and attach to Pods in the specified namespace in order to function.

-- Connecting to the Kubernetes API

You need create a role and binding to the user gitlab-sa.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: gitlab
  name: gitlab-runner-executor
rules:
- apiGroups: [""]
  resources: ["pods", "pods/attach"]
  verbs: ["create", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-runner
  namespace: gitlab
subjects:
- kind: User
  name: gitlab-sa
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: gitlab-runner-executor
  apiGroup: rbac.authorization.k8s.io

Normally the ServiceAccount gitlab-sa is used by Gitlab Runner within the Kubernetes cluster.

It's maybe not necessary when running Gitlab Runner externally to the Cluster, I'm not very sure.

-- silverfox
Source: StackOverflow
10/15/2017

Your client certificate has a usage for server auth instead of client auth. For your TLS client to use the certificate to authenticate to the Kubernetes apiserver it needs to provide a certificate with the client auth usage.

-- Robert Bailey
Source: StackOverflow