Pod cannot pull image from private docker registry

10/29/2019

I am having some real trouble getting my pods to pull images from a private docker registry that I have setup and am able to authenticate to (I can do docker login https://my.website.com/ and I get Login Succeeded without having to put in my username:password) (I am able to run docker pull my.website.com:5000/human/forum and see all the layers being downloaded.) .

I use https://github.com/bazelbuild/rules_k8s#aliasing-eg-k8s_deploy where I specify the namespace to be "default".

I made sure to put "HTTPS://my.website.com:5000/V2/" (in lowercase) in the auth section in the docker config file before I generated the regcred secret.

Notice that I specify the imagePullSecrets below:

# deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: angular-bazel-example-prod
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: angular-bazel-example-prod
    spec:
      containers:
       - name: angular-bazel-example
         image: human/forum:dev
         imagePullPolicy: Always
         ports:
         - containerPort: 8080
      imagePullSecrets:
       - name: regcred # Notice

I made sure to update my certificate authority certificates:

cp /etc/docker/certs.d/my.website.com\:5000/ca.crt /usr/local/share/ca-certificates/my.website.registry.com/
sudo update-ca-certificates

I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/my.website.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/_catalog

> {"repositories":["human/forum"]}

I see sudo curl --user testuser:testpassword --cacert /usr/local/share/ca-certificates/mywebsite.registry.com/ca.crt -X GET https://mywebsite.com:5000/v2/human/forum/tags/list

> {"name":"a/repository","tags":["dev"]}

There must be a way to troubleshoot this but I don't know how.

One thing I am curious about is

kubectl describe pod my-first-pod...
...
Volumes:
  default-token-mtz9g:
    Type:        Secret

Where can I find this volume? I can't kubectl exec into a container because none is running.. because the pod can't pull the image.

Do you have any ideas on how I could troubleshoot this?

Thank you!

Slackware

-- Slackware
kubernetes

1 Answer

10/29/2019

Create a kubernetes secret to access the custom repository. One way is to provide the server, user and password manually. Documentation link.

kubectl create secret docker-registry $YOUR_REGISTRY_NAME --docker-server=https://$YOUR_SERVER_DNS/v2/ --docker-username=$YOUR_USER --docker-password=$YOUR_PASSWORD --docker-email=whatever@gmail.com --namespace=default

Then use it in your yaml

...
      imagePullSecrets:
       - name: $YOUR_REGISTRY_NAME

Regarding the default-token that you see mounted, it belongs to the service account that your pod uses to talk to the kubernetes api. You can find the service account by running kubectl get sa and kubectl describe sa $DEFAULT_SERVICE_ACCOUNT_ID. Find the token by running kubectl get secrets and kubectl describe secret $SECRET_ID. To clarify this service account and token have nothing to do with the docker registry unless you specify it. To include the registry in the service account follow this guide link

-- Rodrigo Loza
Source: StackOverflow