K
Q

Pulling container image from GitLab private registry fails after upgrading to Kubernetes v1.6

6/2/2017

I've setup a Kubenetes cluster to pull container images from the registry of a GitLab private repository.

When I build an image using gitlab-ci.yml - I can use this to successfully login to the registry:

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

When deploying to my Kubernetes cluster, I generate a secret:

- kubectl create secret -n $KUBE_NAMESPACE docker-registry gitlab-registry \
  --docker-server="$CI_REGISTRY" \
  --docker-username="$CI_REGISTRY_USER" \
  --docker-password="$CI_REGISTRY_PASSWORD" \
  --docker-email="$GITLAB_USER_EMAIL" \
  -o yaml --dry-run | kubectl replace -n $KUBE_NAMESPACE --force -f -

I then refer to this secret in my Deployment script using imagePullSecrets. This worked fine when my cluster was v1.5. However, since I upgraded to v1.6 I have not been able to pull a container image.

The error I get in my Kubernetes console is:

Failed to pull image "registry.gitlab.com/user/repo:branch": 
rpc error: code = 2 desc = unauthorized: HTTP Basic: Access denied
Error syncing pod, skipping: failed to "StartContainer" for "helm-chart" with 
ErrImagePull: "rpc error: code = 2 desc = unauthorized: HTTP Basic: Access denied"

To test, I tried running the following on my own PC:

docker login registry.gitlab.com

When I enter my GitLab username and password, everything is fine. However, if I try the following:

docker login \
  -u "gitlab-ci-token" \
  -p "the-auth-code-in-my-secret" \
  registry.gitlab.com

I get the following error:

Error response from daemon: Get https://registry.gitlab.com/v2/:
unauthorized: HTTP Basic: Access denied

I'm not sure if this is a GitLab issue or a Kubernetes issue. Am I just doing something wrong?

-- Mitkins
gitlab
kubernetes

Similar Questions

mapping values are not allowed in this context
mounting vSphere store in Kubernetes pod not working
Kubernetes configMap - only one file
how to get a node's private IP using kubernetes go-client?

1 Answer

6/4/2017

As it turns out, this was a timing issue. My deployment script was creating Kubernetes objects and falling through without waiting. Because this worked in the past, I assumed that $CI_REGISTRY_PASSWORD was still valid after the deployment script finished.

I believe the fact that it worked was just coincidence (my Kubernetes cluster must have pulled the container before the CI pipeline had cleaned up its resources). After adding the following line to `gitlab-ci.yml':

- kubectl rollout status -n "$KUBE_NAMESPACE" -w "deployment/name"

I no longer receive the "HTTP Basic: Access denied" message

-- Mitkins
Source: StackOverflow