No Such Host: Docker daemon can't access kubernetes registry but wget on the same node can connect to the registry

12/29/2019

I have an Alpine Linux based node on a single node kubernetes cluster(for testing). I have a private docker registry installed within my cluster at docker-registry.default:5000. I can login to the alpine node and use wget and access my private docker registry.

kubectl exec -it pod/nuclio-dashboard-5c5c48947b-lpgx8 -- /bin/sh
/ # wget -qO- https://docker:mypassword@docker-registry.default:5000/v2/_catalog
{"repositories":["nuclio/processor-helloworld3"]}

But I can't seem to access it using docker on the same pod. Both Client and Server are 2019 builds

kubectl exec -it pod/nuclio-dashboard-5c5c48947b-lpgx8 -- /bin/sh

/ # which docker
/usr/local/bin/docker
/ # docker login -u docker -p mypassword docker-registry.default:5000
Error response from daemon: Get https://docker-registry.default:5000/v2/: dial tcp: lookup docker-registry.default on 169.254.169.254:53: no such host

I can logon to the Docker Hub registry.

docker login -u my_hub_user  -p my_hub_password
Login Succeeded

EDIT:

On kubectl describe pod nuclio-dashboard-5c5c48947b-lpgx8, we get.

kd pod/nuclio-dashboard-5c5c48947b-2dpnz
Name:           nuclio-dashboard-5c5c48947b-2dpnz
Namespace:      nuclio
Priority:       0
Node:           gke-your-first-cluster-1-pool-1-fe915942-506h/10.128.0.30
Start Time:     Tue, 31 Dec 2019 09:39:45 -0500
Labels:         app=nuclio
                nuclio.io/app=dashboard
                nuclio.io/class=service
                nuclio.io/name=nuclio-dashboard
                pod-template-hash=5c5c48947b
                release=nuclio
Annotations:    nuclio.io/version: 1.3.4-amd64
Status:         Running
IP:             10.4.0.9
Controlled By:  ReplicaSet/nuclio-dashboard-5c5c48947b
Containers:
  nuclio-dashboard:
    Container ID:   docker://4f358607618f89da911e191226313193e38ed5335a3e46c207eee16669f1dd46
    Image:          quay.io/nuclio/dashboard:1.3.4-amd64
    Image ID:       docker-pullable://quay.io/nuclio/dashboard@sha256:e6d94f7bf46601b2454a9e73ba292c62edac3d4684ea15057855af2277eab8a5
    Port:           8070/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 31 Dec 2019 09:40:27 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      NUCLIO_DASHBOARD_REGISTRY_URL:                <set to the key 'registry_url' of config map 'nuclio-registry-url'>  Optional: true
      NUCLIO_DASHBOARD_DEPLOYMENT_NAME:             nuclio-dashboard
      NUCLIO_CONTAINER_BUILDER_KIND:                docker
      NUCLIO_DASHBOARD_EXTERNAL_IP_ADDRESSES:
      NUCLIO_DASHBOARD_HTTP_INGRESS_HOST_TEMPLATE:
    Mounts:
      /etc/nuclio/dashboard/registry-credentials from registry-credentials (ro)
      /var/run/docker.sock from docker-sock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from nuclio-nuclio-token-d7fwp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:
  registry-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nuclio-registry-credentials
    Optional:    true
  nuclio-nuclio-token-d7fwp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nuclio-nuclio-token-d7fwp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
-- RAbraham
alpine
docker
kubernetes

1 Answer

12/29/2019

Kubernetes will inject the internal DNS servers to the pod's /etc/resolv.conf file. That is why you can access the registry from Pod.

Usually, this DNS service will not be exposed outside of Pod network.

When you use the docker command, you are inside the host and the host will be pointing to a different DNS server that can't solve the internal service name of the registry.

To access the registry from your host, you need below.

1) Expose the registry Service as NodePort or LoadBalancer

(As you are in a test environment, use NodePort)doc link

2) Create proper DNS entry to resolve the name to IP (here IP will be the Node's IP incase of NodePort service). As you have only one node, create an entry in /etc/hosts file to resolve the registry FQDN.

-- Ansil
Source: StackOverflow