Kubernetes: Pull image from ClusterIP private registry?

11/3/2020

I've been working on a setup for Kubernetes (while simultaneously learning Kubernetes) and I've hit a snag that I thing suggests what I want to do is impossible. My setup is described in full detail here: https://stackoverflow.com/questions/64635307/can-a-self-signed-cert-secure-multiple-cns-fqdns

After generated a self-signed cert with an SAN and configuring my registry and setting up a DaemonSet to trust the self-signed certificate on all nodes in my cluster, I ran into a new issue: Kubernetes coudn't find the host "docker-registry"

There is a ClusterIP service running with the name "docker-registry" which points to the Docker registry. When I kubectl exec a shell on a pod in the cluster for debug purposes I can see that the KubeDNS is working and locating this service:

root@debug:/# ping docker-registry
PING docker-registry.default.svc.cluster.local (10.245.46.24) 56(84) bytes of data.
^C
--- docker-registry.default.svc.cluster.local ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2039ms

I assume the packet loss is because the service only responds to traffic on port 443 and doesn't respond to pings. What's important is that the hostname was resolved to an IP address.

Here's my problem, though: This hostname only resolves from pods within the cluster. Kubernetes, itself, can't hit this hostname. I've defined a pod like so:

apiVersion: v1
kind: Pod
metadata:
  name: test-docker
  labels:
    name: test
spec:
  containers:
  - name: test
    image: docker-registry/my-ubuntu

When I try to create this pod (kubectl create -f pods/test.yml) I get the following error:

...
Failed to pull image "docker-registry/my-ubuntu": rpc error: code = Unknown desc = Error response from daemon: pull access denied for docker-registry/my-ubuntu, repository does not exist or may require 'docker login'
...

This makes some sense, intuitively. While other pods can access my registry at the host docker-registry, the Kubernetes nodes can't because this isn't a valid host or defined in the /etc/hosts file.

I tried doing some preliminary Googling and found the following Kubernetes document: https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/

This suggests that the only ways to access a service running inside of a cluster (as is the case with my Docker registry) are:

  1. Use a NodePort (this is no good, as it would expose my Docker registry to the internet)
  2. Use the Proxy Verb (it's not clear to me exactly how this method works or how authentication is handled, but it sounds like I may be able to replace docker-registry with a ridiculously long and hard-to-remember URL like https://<node-ip>/api/v1/namespaces/default/services/docker-registry)
  3. Access from a pod within the cluster (this is no good, as I need Kubernetes to perform the image pull and I'm not doing it manually)

It may not be possible to accomplish what I want (have a Docker registry that's available to any node in my cluster but not available to the public internet) but I wanted to ask on StackOverflow if there's anything I'm missing.

If what I want isn't possible then my last resort (before paying for Docker Hub) will be to try just exposing my registry to the public internet but utilize Docker authentication and a pull secret (plus possibly putting my registry behind a firewall to block traffic from outside of my cluster)

-- stevendesu
docker
docker-registry
kubernetes

0 Answers