K
Q

How to fix k8s namespace permissions in gitlab ci

6/14/2019

As I'm playing around with K8s deployment and Gitlab CI my deployment got stuck with the state ContainerStarting.

To reset that, I deleted the K8s namespace using kubectl delete namespaces my-namespace.

Now my Gitlab runner shows me

$ ensure_namespace
Checking namespace [MASKED]-docker-3
error: the server doesn't have a resource type "namespace"
error: You must be logged in to the server (Unauthorized)

I think that has something to do with RBAC and most likely Gitlab created that namespace with some arguments and permissions (but I don't know exactly when and how that happens), which are missing now because of my deletion.

Anybody got an idea on how to fix this issue?

-- Hans Höchtl
gitlab-ci
kubernetes

Similar Questions

Best practices for managing multiple environments with Helm
Use python client to create Kubernetes CronJob from yaml
How to get filebeat to ignore certain container logs
How to install kube-dns on minikube?
Get all the pods dynamically created belonging to an autoscaled deployment

3 Answers

10/6/2019

In my case I had to delete the namespace in Gitlab database, so gitlab would readd service account and namespace:

On the gitlab machine or task runner enter the PostgreSQL console:

gitlab-rails dbconsole -p

Then select the database:

\c gitlabhq_production

Next step is to find the namespace that was deleted:

SELECT id, namespace FROM clusters_kubernetes_namespaces;

Take the id of the namespace to delete it:

DELETE FROM clusters_kubernetes_namespaces WHERE id IN (6,7);

Now you can restart the pipeline and the namespace and service account will be readded.

-- scasei
Source: StackOverflow
6/15/2019

Deleting the namespace manually caused the necessary secrets from Gitlab to get removed. It seems they get autocreated on the first ever deployment and it's impossible to repeat that process.

I had to create a new repo and push to it. Now everything works.

-- Hans Höchtl
Source: StackOverflow
6/27/2019

Another solution is removing the cluster from Gitlab (under operations/kubernetes in your repo) and re-adding it.

-- gtakevin
Source: StackOverflow