I'm learning Kubernetes and want to set up a Docker registry to run within my cluster, deploy any custom code to this private registry, then have my nodes pull images from this private registry to create pods. I've described my setup in this StackOverflow question

Originally I was caught up trying to figure out SSL certificates, but for now I've postponed that and I'm trying to work with an insecure registry. To that end I've created the following pod to run my registry (I know it's a pod and not a replica set or deployment -- this is only for experimental purposes and I'll make it cleaner once it's working):

apiVersion: v1
kind: Pod
metadata:
  name: docker-registry
  labels:
    app: docker-registry
spec:
  containers:
    - name: docker-registry
      image: registry:2
      ports:
        - containerPort: 80
          hostPort: 80
      env:
        - name: REGISTRY_HTTP_ADDR
          value: 0.0.0.0:80

I then created the following NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: docker-registry-external
  labels:
    app: docker-registry
spec:
  type: NodePort
  ports:
    - targetPort: 80
      port: 80
      nodePort: 32000
  selector:
    app: docker-registry

I have a load balancer set up in front of my Kubernetes cluster which I configured to route traffic on port 80 to port 32000. So I can hit this registry at http://example.com

I then updated my local /etc/docker/daemon.json as follows:

{
  "insecure-registries": ["example.com"]
}

With this I was able to push an image to my registry successfully:

> docker pull ubuntu
> docker tag ubuntu example.com/my-ubuntu
> docker push exapmle.com/my-ubuntu
The push refers to repository [example.com/my-ubuntu]
cc9d18e90faa: Pushed 
0c2689e3f920: Pushed 
47dde53750b4: Pushed 
latest: digest: sha256:1d7b639619bdca2d008eca2d5293e3c43ff84cbee597ff76de3b7a7de3e84956 size: 943

Now I want to try and pull this image when creating a pod. So I created the following ClusterIP service to make my registry accessible within my cluster:

apiVersion: v1
kind: Service
metadata:
  name: docker-registry-internal
  labels:
    app: docker-registry
spec:
  type: ClusterIP
  ports:
    - targetPort: 80
      port: 80
  selector:
    app: docker-registry

Then I created a secret:

apiVersion: v1
kind: Secret
metadata:
  name: local-docker
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewoJImluc2VjdXJlLXJlZ2lzdHJpZXMiOiBbImRvY2tlci1yZWdpc3RyeS1pbnRlcm5hbCJdCn0K

The base64 bit decodes to:

{
	"insecure-registries": ["docker-registry-internal"]
}

Finally, I created the following pod:

apiVersion: v1
kind: Pod
metadata:
  name: test-docker
  labels:
    name: test
spec:
  imagePullSecrets:
    - name: local-docker
  containers:
  - name: test
    image: docker-registry-internal/my-ubuntu

When I tried to create this pod (kubectl create -f test-pod.yml) and looked at my cluster, this is what I saw:

> kubectl get pods
NAME             READY  STATUS        RESTARTS  AGE
test-docker      0/1    ErrImagePull  0         4s
docker-registry  1/1    Running       0         34m
> kubectl describe pod test-docker
...
...
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       3m33s                  default-scheduler  Successfully assigned default/test-docker to pool-uqa-dev-3sli8
  Normal   Pulling         3m22s (x2 over 3m32s)  kubelet            Pulling image "docker-registry-internal/my-ubuntu"
  Warning  Failed          3m22s (x2 over 3m32s)  kubelet            Failed to pull image "docker-registry-internal/my-ubuntu": rpc error: code = Unknown desc = Error response from daemon: pull access denied for docker-registry-internal/my-ubuntu, repository does not exist or may require 'docker login'
  Warning  Failed          3m22s (x2 over 3m32s)  kubelet            Error: ErrImagePull
  Normal   SandboxChanged  3m19s (x7 over 3m32s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         3m18s (x6 over 3m30s)  kubelet            Back-off pulling image "docker-registry-internal/my-ubuntu"
  Warning  Failed          3m18s (x6 over 3m30s)  kubelet            Error: ImagePullBackOff

It's clearly failing to find the host "docker-registry-internal", despite the ClusterIP service.

I tried inspecting a pod from the inside using a trick I found online:

> kubectl run -i --tty --rm debug --image=ubuntu --restart=Never -- bash
If you don't see a command prompt, try pressing enter.
root@debug:/# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.244.1.67     debug

It doesn't seem like ClusterIP services are being added to the /etc/hosts file, so I'm not sure how services are supposed to find one another?

I tried watching several Kubernetes tutorials on general service communication (e.g. an app pod communicating with a redis pod) and every time all they did was supply the service name as a host and it magically connected. I'm not sure if I'm missing something. Bear in mind I'm brand new to Kubernetes so the internals are still mystical to me.