I have created a k8 cluster in GKE.
I have a docker registry created in Artifactory, this artifactory is hosted on AWS. I have a route53 entry for docker-repo.aws.abc.com in aws.abc.com Hosted zone in AWS
Now, I need to configure my cluster so that the docker images are pulled from artifactory.
I went through documentation and understand I will have to add stubDomain in my kube-dns configmaps.
kubectl edit cm kube-dns -n kube-system
apiVersion: v1
data:
stubDomains: |
{"aws.abc.com" : ["XX.XX.XX.XX"]}
kind: ConfigMap
metadata:
creationTimestamp: 2019-05-21T14:30:15Z
labels:
addonmanager.kubernetes.io/mode: EnsureExists
name: kube-dns
namespace: kube-system
resourceVersion: "7669"
selfLink: /api/v1/namespaces/kube-system/configmaps/kube-dns
uid: f378aa5f-7bd4-11e9-9df2-42010aa93d03However, still docker pull command fails.
docker pull docker-repo.aws.abc.com/abc-sampleapp-java/abc-service:V-57bc9c9-201
Error response from daemon: Get https://docker-repo.aws.abc.com/v2/: dial tcp: lookup docker-dev-repo.aws.abc.com on 169.254.169.254:53: no such host
Note: When I make an entry in /etc/hosts file on worker nodes, docker pull works fine.
Pulling an image from registry on pod start uses different DNS settings than when we call DNS from pods inside a cluster.
When Kubernetes starts new pod, it schedules it to the node and then agent on the node named kubelet calls container engine (Docker) to download an image and run it with designed configuration.
Docker uses system DNS to resolve the address of a registry, because it works right on our host system, not in the Kubernetes, that is why any DNS settings will not affect DNS resolving on the image downloading stage. https://github.com/kubernetes/kubernetes/issues/8735 is a discussion about it on Github.
If we want to change DNS settings and override the registry IP to use it on image downloading stage, we should set it in the host system. In the configuration, we need to modify DNS settings on all your nodes in the cluster. The simplest way to do it is using /etc/hosts file and adding a record with your custom IP, e.g. 192.168.1.124 example.com.
After that modifications, Docker on nodes will use the record from /etc/hosts for your registry instead of global DNS records, because that file has higher priority and you will be able to run pods with your image.
To update the host file. you can use a DeamonSet with Security Context as privileged see below:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: hosts-fix-script
namespace: kube-system
labels:
app: hosts-fix-script
spec:
selector:
matchLabels:
name: hosts-fix
template:
metadata:
labels:
name: hosts-fix
spec:
hostPID: true
containers:
- name: hosts-fix-script
image: gcr.io/google-containers/startup-script:v1
imagePullPolicy: Always
securityContext:
privileged: true
env:
- name: STARTUP_SCRIPT
value: |
#!/bin/bash
echo "10.0.0.11 onprem.registry" >> /etc/hosts
echo 'Done'you need to run the kubectl apply -f