kubernetes dones't reach internal registry

4/16/2020

I've deployed an docker registry inside my kubernetes:

$ kubectl get service
NAME                       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
registry-docker-registry   ClusterIP   10.43.39.81   <none>        443/TCP   162m

I'm able to pull images from my machine (service is exposed via an ingress rule):

$ docker pull registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty@sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd
...
Status: Downloaded newer image for registry-do...

When I'm trying to test it in order to deploy my image into the same kubernetes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: covid-backend
  namespace: skaffold
spec:
  replicas: 3
  selector:
    matchLabels:
      app: covid-backend
  template:
    metadata:
      labels:
        app: covid-backend
    spec:
      containers:
      - image: registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty@sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd
        name: covid-backend
        ports:
        - containerPort: 8080

Then, I've tried to deploy it:

$ cat pod.yaml | kubectl apply -f -

However, kubernetes isn't able to reach registry:

Extract of kubectl get events:

6s          Normal    Pulling             pod/covid-backend-774bd78db5-89vt9    Pulling image "registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty@sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd"
1s          Warning   Failed              pod/covid-backend-774bd78db5-89vt9    Failed to pull image "registry-docker-registry.registry/skaffold-covid-backend:c5dfd81-dirty@sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd": rpc error: code = Unknown desc = failed to pull and unpack image "registry-docker-registry.registry/skaffold-covid-backend@sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd": failed to resolve reference "registry-docker-registry.registry/skaffold-covid-backend@sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd": failed to do request: Head https://registry-docker-registry.registry/v2/skaffold-covid-backend/manifests/sha256:76312ebc62c4b3dd61b4451fe01b1ecd2e6b03a2b3146c7f25df3d3cfb4512cd: dial tcp: lookup registry-docker-registry.registry: Try again
1s          Warning   Failed              pod/covid-backend-774bd78db5-89vt9    Error: ErrImagePull

As you can see, kubernetes is not able to get access to the internal deployed registry...

Any ideas?

-- Jordi
docker-registry
kubernetes

1 Answer

4/20/2020

I would recommend to follow docs from k3d, they are here.

More precisely this one

Using your own local registry

If you don't want k3d to manage your registry, you can start it with some docker commands, like:

docker volume create local_registry
docker container run -d --name registry.local -v local_registry:/var/lib/registry --restart always -p 5000:5000 registry:2

These commands will start you registry in registry.local:5000. In order to push to this registry, you will need to add the line at /etc/hosts as we described in the previous section . Once your registry is up and running, we will need to add it to your registries.yaml configuration file. Finally, you must connect the registry network to the k3d cluster network: docker network connect k3d-k3s-default registry.local. And then you can check you local registry.

Pushing to your local registry address

The registry will be located, by default, at registry.local:5000 (customizable with the --registry-name and --registry-port parameters). All the nodes in your k3d cluster can resolve this hostname (thanks to the DNS server provided by the Docker daemon) but, in order to be able to push to this registry, this hostname but also be resolved from your host.

The easiest solution for this is to add an entry in your /etc/hosts file like this:

127.0.0.1 registry.local

Once again, this will only work with k3s >= v0.10.0 (see the section below when using k3s <= v0.9.1)

Local registry volume

The local k3d registry uses a volume for storying the images. This volume will be destroyed when the k3d registry is released. In order to persist this volume and make these images survive the removal of the registry, you can specify a volume with the --registry-volume and use the --keep-registry-volume flag when deleting the cluster. This will create a volume with the given name the first time the registry is used, while successive invocations will just mount this existing volume in the k3d registry container.

Docker Hub cache

The local k3d registry can also be used for caching images from the Docker Hub. You can start the registry as a pull-through cache when the cluster is created with --enable-registry-cache. Used in conjuction with --registry-volume/--keep-registry-volume can speed up all the downloads from the Hub by keeping a persistent cache of images in your local machine.

Testing your registry

You should test that you can

  • push to your registry from your local development machine.
  • use images from that registry in Deployments in your k3d cluster.

We will verify these two things for a local registry (located at registry.local:5000) running in your development machine. Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker's documentation for this).

Firstly, we can download some image (like nginx) and push it to our local registry with:

docker pull nginx:latest
docker tag nginx:latest registry.local:5000/nginx:latest
docker push registry.local:5000/nginx:latest

Then we can deploy a pod referencing this image to your cluster:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-registry
  labels:
    app: nginx-test-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test-registry
  template:
    metadata:
      labels:
        app: nginx-test-registry
    spec:
      containers:
      - name: nginx-test-registry
        image: registry.local:5000/nginx:latest
        ports:
        - containerPort: 80
EOF

Then you should check that the pod is running with kubectl get pods -l "app=nginx-test-registry".


Additionaly there are 2 github links worth visting

-- jt97
Source: StackOverflow